[petsc-users] pcfieldsplit for a composite dm with multiple subfields
Barry Smith
bsmith at mcs.anl.gov
Tue Sep 8 20:27:18 CDT 2015
Did you add the line I told you to?
> On Sep 8, 2015, at 8:23 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>
> I just installed a fresh 3.6.1 pull, compiled against it, and I’m still having an error.
> 0 SNES Function norm 1.421454390131e-02
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep 8 21:21:11 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /opt/petsc/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /opt/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 121 in /opt/petsc/src/snes/interface/snesj.c
> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2232 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 231 in /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3906 in /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Vec is locked read only, argument # 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.1, unknown
> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep 8 21:21:11 2015
> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
> [0]PETSC ERROR: #7 VecGetArray() line 1646 in /opt/petsc/src/vec/vec/interface/rvector.c
>
>
>
>
> -gideon
>
>> On Sep 8, 2015, at 8:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>> This is our fault. You need to add a line
>>
>> ierr = MatSetOption(B,MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE);CHKERRQ(ierr);
>>
>> after
>>
>> PetscFunctionBegin;
>>
>> in SNESComputeJacobianDefault()
>>
>> then do
>>
>> make gnumake
>>
>> in that directory.
>>
>> You really should update to PETSc 3.6 (http://www.mcs.anl.gov/petsc/download/index.html note it is best to obtain with git) It will make both your life and our lives easier and there is no downside.
>>
>> Barry
>>
>>> On Sep 8, 2015, at 6:24 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>
>>> When I use that flag, I get this error:
>>>
>>> 0 SNES Function norm 1.421454390131e-02
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Argument out of range
>>> [0]PETSC ERROR: New nonzero at (3,0) caused a malloc
>>> Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Tue Sep 8 19:24:08 2015
>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 487 in /opt/petsc-3.5.4/src/mat/impls/aij/seq/aij.c
>>> [0]PETSC ERROR: #2 MatSetValues() line 1135 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>> [0]PETSC ERROR: #3 SNESComputeJacobianDefault() line 120 in /opt/petsc-3.5.4/src/snes/interface/snesj.c
>>> [0]PETSC ERROR: #4 SNESComputeJacobian() line 2193 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /opt/petsc-3.5.4/src/snes/impls/ls/ls.c
>>> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /opt/petsc-3.5.4/src/snes/interface/snes.c
>>>
>>> -gideon
>>>
>>>> On Sep 8, 2015, at 7:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>
>>>>
>>>> You can run with -snes_fd it will be slow so run with only two levels of grid sequencing; check the convergence rate of Newton on the finer grid. It will hopefully be much better (also KSP should converge in one iteration). Send the output
>>>>
>>>> Barry
>>>>
>>>>> On Sep 8, 2015, at 2:20 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>
>>>>> I went through my code and nothing jumps out at me as clearly wrong. Is there a way to dump to disk vectors and matrices that would be useful for the comparison? I imagine I’d want the jacobian that I am specifying, but what would I want to compare that against in terms of using -snes_mf_operator?
>>>>>
>>>>>
>>>>> -gideon
>>>>>
>>>>>> On Sep 8, 2015, at 11:54 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>
>>>>>> On Tue, Sep 8, 2015 at 10:14 AM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>> Just to clarify, again, during the linear solve phase, even though the PC residual norm was small (getting to 10^{-10} in 2-3 iterations) that the true residual norm could be comparatively large signified there is something wrong. The one thing i was curious about is why it seems that there is only an issue during grid sequencing. I’m not seeing that kind of behavior on the coarsest mesh.
>>>>>>
>>>>>> That definitely sounds like a bug in the Jacobian evaluation. Maybe you assume a certain mesh size, and when
>>>>>> PETSc refines the DA and passes in the new one something goes wrong?
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> -gideon
>>>>>>
>>>>>>> On Sep 7, 2015, at 10:58 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Sep 7, 2015, at 9:35 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>
>>>>>>>> Alright, I’ll take another look at what has been implemented in my Jacobian file. One thing I’m a bit unclear about is do I need these -snes_mf and snes_mf_operator flags if I have, coded in,
>>>>>>>>
>>>>>>>> SNESSetJacobian(snes,NULL,NULL,form_function_jacobian, ctx);
>>>>>>>
>>>>>>> Gideon,
>>>>>>>
>>>>>>> In your case the DM creates that Jacobian matrix object and form_function_jacobian's job is just to fill it up with the correct values for the current solution value. If you use -snes_mf_operator then SNES uses a matrix-free multiple for the first matrix and your Jacobian to build the preconditioner. With -snes_mf it uses a matrix-free multiple for the first and second matrix and ignores what you compute. Since it is matrix free in both it cannot build a preconditioner hence the -pc_type none
>>>>>>>
>>>>>>> Using -snes_mf and -snes_mf_operator are a way of testing if the Jacobian you provide is correct
>>>>>>>
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -gideon
>>>>>>>>
>>>>>>>>> On Sep 7, 2015, at 10:27 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If the Jacobian you provide is correct then using -snes_mf with a PC type of LU should give two iterations (or at most three) to get very good (10^-12) convergence of the linear system. Here you are not getting that after a few SNES iterations. So I am strongly leaning to something wrong with the Jacobian you provide. -snes_type test unfortunately doesn't catch all problems.
>>>>>>>>>
>>>>>>>>> You can try -snes_mf -pc_type none and see how the linear solver converges.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Barry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Sep 7, 2015, at 9:16 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Off the top of my head, I can’t rule out a null space in this problem. I thought I did a pretty good job of coding the Jacobian by hand. And before I made that change to the Jacobian file, changing J for Jpre, if I ran with -snes_check_jacobian, it appeared to be satisfactory.
>>>>>>>>>>
>>>>>>>>>> When I run with -ksp_type fgmres, I get
>>>>>>>>>>
>>>>>>>>>> 0 SNES Function norm 3.857327250267e-09
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.857327250267e-09 true resid norm 3.857327250267e-09 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.490987252203e-13 true resid norm 4.490987252203e-13 ||r(i)||/||b|| 1.164274369485e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.379711327483e-17 true resid norm 5.151515251741e-13 ||r(i)||/||b|| 1.335514183139e-04
>>>>>>>>>> 1 SNES Function norm 2.479697868412e-12
>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.066222213176e+03 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 8.556950606195e-03 true resid norm 8.556950606090e-03 ||r(i)||/||b|| 1.689019992814e-06
>>>>>>>>>> 1 SNES Function norm 8.484696862318e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 8.484696862318e+02 true resid norm 8.484696862318e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.384145175257e-01 true resid norm 1.384145175257e-01 ||r(i)||/||b|| 1.631343108325e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.495757031958e-04 true resid norm 1.605753181683e-01 ||r(i)||/||b|| 1.892528640374e-04
>>>>>>>>>> 2 SNES Function norm 6.551139809777e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 6.551139809777e+02 true resid norm 6.551139809777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.699223578306e-02 true resid norm 5.699223578306e-02 ||r(i)||/||b|| 8.699590825096e-05
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.336876642454e-05 true resid norm 6.919655452070e-02 ||r(i)||/||b|| 1.056252141306e-04
>>>>>>>>>> 3 SNES Function norm 5.771483337231e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.771483337231e+02 true resid norm 5.771483337231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.809574952128e-02 true resid norm 4.809574952128e-02 ||r(i)||/||b|| 8.333342870631e-05
>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.460582114169e-05 true resid norm 6.436764219063e-02 ||r(i)||/||b|| 1.115270346107e-04
>>>>>>>>>> 4 SNES Function norm 5.237540245670e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 5.237540245670e+02 true resid norm 5.237540245670e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.634190283952e-02 true resid norm 5.634190283952e-02 ||r(i)||/||b|| 1.075732122271e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.087123509737e-04 true resid norm 6.710344480682e-02 ||r(i)||/||b|| 1.281201511765e-04
>>>>>>>>>> 5 SNES Function norm 4.753058321803e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.753058321803e+02 true resid norm 4.753058321803e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.480046646239e-02 true resid norm 7.480046646239e-02 ||r(i)||/||b|| 1.573733402750e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.930291355515e-04 true resid norm 8.744803271470e-02 ||r(i)||/||b|| 1.839826629384e-04
>>>>>>>>>> 6 SNES Function norm 4.221482124411e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 4.221482124411e+02 true resid norm 4.221482124411e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.021730633747e-01 true resid norm 1.021730633747e-01 ||r(i)||/||b|| 2.420312590781e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.424059347969e-04 true resid norm 1.186664050361e-01 ||r(i)||/||b|| 2.811012851384e-04
>>>>>>>>>> 7 SNES Function norm 3.803620028964e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.803620028964e+02 true resid norm 3.803620028964e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.347601667746e-01 true resid norm 2.347601667746e-01 ||r(i)||/||b|| 6.172019418000e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.585189594479e-03 true resid norm 2.804292032643e-01 ||r(i)||/||b|| 7.372692359618e-04
>>>>>>>>>> 8 SNES Function norm 3.769519577570e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.769519577570e+02 true resid norm 3.769519577570e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.473450825400e+00 true resid norm 2.473450825400e+00 ||r(i)||/||b|| 6.561713699852e-03
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.512219713818e-02 true resid norm 2.900441161617e+00 ||r(i)||/||b|| 7.694458410235e-03
>>>>>>>>>> 3 KSP unpreconditioned resid norm 9.156193230828e-04 true resid norm 2.963280717470e+00 ||r(i)||/||b|| 7.861162825903e-03
>>>>>>>>>> 9 SNES Function norm 3.766389203406e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.766389203406e+02 true resid norm 3.766389203406e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.885345065329e+00 true resid norm 3.885345065329e+00 ||r(i)||/||b|| 1.031583528812e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.541366781733e-03 true resid norm 4.481523065039e+00 ||r(i)||/||b|| 1.189872533881e-02
>>>>>>>>>> 10 SNES Function norm 3.751822489648e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.751822489648e+02 true resid norm 3.751822489648e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.504123802743e-01 true resid norm 5.504123802743e-01 ||r(i)||/||b|| 1.467053363513e-03
>>>>>>>>>> 2 KSP unpreconditioned resid norm 3.266605004942e-03 true resid norm 6.521599031098e-01 ||r(i)||/||b|| 1.738248291089e-03
>>>>>>>>>> 11 SNES Function norm 3.737189963925e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737189963925e+02 true resid norm 3.737189963925e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 7.988643057771e+00 true resid norm 7.988643057771e+00 ||r(i)||/||b|| 2.137606901143e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 4.764871173758e-03 true resid norm 9.352795885660e+00 ||r(i)||/||b|| 2.502627903837e-02
>>>>>>>>>> 3 KSP unpreconditioned resid norm 3.282807486232e-04 true resid norm 9.151099849649e+00 ||r(i)||/||b|| 2.448657932292e-02
>>>>>>>>>> 12 SNES Function norm 3.737110331594e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.737110331594e+02 true resid norm 3.737110331594e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 3.129801991140e-01 true resid norm 3.129801991140e-01 ||r(i)||/||b|| 8.374925312429e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.125627067813e-03 true resid norm 3.626216163131e-01 ||r(i)||/||b|| 9.703262257137e-04
>>>>>>>>>> 13 SNES Function norm 3.684175513154e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.684175513154e+02 true resid norm 3.684175513154e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.242155815449e-01 true resid norm 1.242155815449e-01 ||r(i)||/||b|| 3.371597827014e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.368849704689e-04 true resid norm 1.448989111108e-01 ||r(i)||/||b|| 3.933007822060e-04
>>>>>>>>>> 14 SNES Function norm 3.509930783933e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.509930783933e+02 true resid norm 3.509930783933e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.839683743117e-02 true resid norm 1.839683743118e-02 ||r(i)||/||b|| 5.241367583483e-05
>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.044153843783e-06 true resid norm 2.242253151407e-02 ||r(i)||/||b|| 6.388311591987e-05
>>>>>>>>>> 15 SNES Function norm 3.162121096565e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 3.162121096565e+02 true resid norm 3.162121096565e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 2.154093218732e-02 true resid norm 2.154093218732e-02 ||r(i)||/||b|| 6.812178132811e-05
>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.294811898520e-06 true resid norm 2.535893281680e-02 ||r(i)||/||b|| 8.019595721477e-05
>>>>>>>>>> 16 SNES Function norm 2.890165052818e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.890165052818e+02 true resid norm 2.890165052818e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.746945848272e-02 true resid norm 6.746945848272e-02 ||r(i)||/||b|| 2.334450014089e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 6.877255989792e-05 true resid norm 7.701022876470e-02 ||r(i)||/||b|| 2.664561620438e-04
>>>>>>>>>> 17 SNES Function norm 2.816929421445e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.816929421445e+02 true resid norm 2.816929421445e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.895856197699e-01 true resid norm 4.895856197699e-01 ||r(i)||/||b|| 1.738011666329e-03
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.103295403077e-02 true resid norm 5.924581730391e-01 ||r(i)||/||b|| 2.103205598723e-03
>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.552844644077e-05 true resid norm 6.041340389186e-01 ||r(i)||/||b|| 2.144654510401e-03
>>>>>>>>>> 18 SNES Function norm 2.814407559738e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.814407559738e+02 true resid norm 2.814407559738e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 1.709056815086e-01 true resid norm 1.709056815086e-01 ||r(i)||/||b|| 6.072527801359e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 7.267004791015e-04 true resid norm 2.045719509609e-01 ||r(i)||/||b|| 7.268739392527e-04
>>>>>>>>>> 19 SNES Function norm 2.810261821835e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.810261821835e+02 true resid norm 2.810261821835e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.414127232289e-02 true resid norm 6.414127232290e-02 ||r(i)||/||b|| 2.282394893762e-04
>>>>>>>>>> 2 KSP unpreconditioned resid norm 1.134804652646e-04 true resid norm 7.350062521045e-02 ||r(i)||/||b|| 2.615436919058e-04
>>>>>>>>>> 20 SNES Function norm 2.755723672077e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755723672077e+02 true resid norm 2.755723672077e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.096136080591e+00 true resid norm 6.096136080591e+00 ||r(i)||/||b|| 2.212172483896e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 9.111133958404e-01 true resid norm 6.649792338356e+00 ||r(i)||/||b|| 2.413083868218e-02
>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.910677194353e-01 true resid norm 6.843103003287e+00 ||r(i)||/||b|| 2.483232652325e-02
>>>>>>>>>> 4 KSP unpreconditioned resid norm 4.788261382489e-02 true resid norm 6.760631408983e+00 ||r(i)||/||b|| 2.453305270585e-02
>>>>>>>>>> 5 KSP unpreconditioned resid norm 1.390068885512e-02 true resid norm 6.776042197126e+00 ||r(i)||/||b|| 2.458897554129e-02
>>>>>>>>>> 6 KSP unpreconditioned resid norm 2.384477406046e-03 true resid norm 6.655789279547e+00 ||r(i)||/||b|| 2.415260044753e-02
>>>>>>>>>> 21 SNES Function norm 2.755720389981e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755720389981e+02 true resid norm 2.755720389981e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 5.357900039856e+00 true resid norm 5.357900039856e+00 ||r(i)||/||b|| 1.944282903060e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 5.929328578269e-01 true resid norm 5.798140540809e+00 ||r(i)||/||b|| 2.104038044603e-02
>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.086022506600e-01 true resid norm 5.986795329019e+00 ||r(i)||/||b|| 2.172497380643e-02
>>>>>>>>>> 4 KSP unpreconditioned resid norm 1.109330548287e-02 true resid norm 5.917768589366e+00 ||r(i)||/||b|| 2.147448852532e-02
>>>>>>>>>> 5 KSP unpreconditioned resid norm 3.644882603935e-03 true resid norm 6.000976794018e+00 ||r(i)||/||b|| 2.177643572198e-02
>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.950365335797e-04 true resid norm 5.979808554437e+00 ||r(i)||/||b|| 2.169962009272e-02
>>>>>>>>>> 22 SNES Function norm 2.755716464881e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755716464881e+02 true resid norm 2.755716464881e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.907844411256e+00 true resid norm 4.907844411256e+00 ||r(i)||/||b|| 1.780968569808e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.044873964073e-01 true resid norm 5.843433224469e+00 ||r(i)||/||b|| 2.120476942726e-02
>>>>>>>>>> 3 KSP unpreconditioned resid norm 4.734069809407e-02 true resid norm 5.888726835640e+00 ||r(i)||/||b|| 2.136913180542e-02
>>>>>>>>>> 4 KSP unpreconditioned resid norm 2.325616228250e-03 true resid norm 5.877722539415e+00 ||r(i)||/||b|| 2.132919919129e-02
>>>>>>>>>> 23 SNES Function norm 2.755713163168e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755713163168e+02 true resid norm 2.755713163168e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 6.846535958070e+00 true resid norm 6.846535958070e+00 ||r(i)||/||b|| 2.484487881242e-02
>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.481071550890e+00 true resid norm 8.756034834835e+00 ||r(i)||/||b|| 3.177411550616e-02
>>>>>>>>>> 3 KSP unpreconditioned resid norm 8.153960511371e-01 true resid norm 1.077766099902e+01 ||r(i)||/||b|| 3.911024247035e-02
>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.272907968554e-01 true resid norm 1.158585973964e+01 ||r(i)||/||b|| 4.204305402497e-02
>>>>>>>>>> 5 KSP unpreconditioned resid norm 2.568557800847e-01 true resid norm 1.111425590026e+01 ||r(i)||/||b|| 4.033168636275e-02
>>>>>>>>>> 6 KSP unpreconditioned resid norm 9.535234538303e-02 true resid norm 1.129251830278e+01 ||r(i)||/||b|| 4.097856937258e-02
>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.803098494703e-03 true resid norm 1.144578323652e+01 ||r(i)||/||b|| 4.153474095018e-02
>>>>>>>>>> 8 KSP unpreconditioned resid norm 2.151080788386e-04 true resid norm 1.140343808626e+01 ||r(i)||/||b|| 4.138107782289e-02
>>>>>>>>>> 24 SNES Function norm 2.755712363532e+02
>>>>>>>>>> 0 KSP unpreconditioned resid norm 2.755712363532e+02 true resid norm 2.755712363532e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>> 1 KSP unpreconditioned resid norm 4.232622501221e+01 true resid norm 4.232622501221e+01 ||r(i)||/||b|| 1.535944954645e-01
>>>>>>>>>> 2 KSP unpreconditioned resid norm 2.491207938618e+01 true resid norm 6.073593758143e+01 ||r(i)||/||b|| 2.204001345902e-01
>>>>>>>>>> 3 KSP unpreconditioned resid norm 1.873650490509e+01 true resid norm 8.524306125765e+01 ||r(i)||/||b|| 3.093322161838e-01
>>>>>>>>>> 4 KSP unpreconditioned resid norm 5.309932679768e+00 true resid norm 1.074506397563e+02 ||r(i)||/||b|| 3.899196490108e-01
>>>>>>>>>> 5 KSP unpreconditioned resid norm 4.024003774177e+00 true resid norm 1.106165625327e+02 ||r(i)||/||b|| 4.014082311221e-01
>>>>>>>>>> 6 KSP unpreconditioned resid norm 3.757785755885e+00 true resid norm 1.126119690688e+02 ||r(i)||/||b|| 4.086492137535e-01
>>>>>>>>>> 7 KSP unpreconditioned resid norm 3.609856365266e+00 true resid norm 1.153615338633e+02 ||r(i)||/||b|| 4.186269052965e-01
>>>>>>>>>> 8 KSP unpreconditioned resid norm 3.599697975830e+00 true resid norm 1.145508733644e+02 ||r(i)||/||b|| 4.156851595993e-01
>>>>>>>>>> 9 KSP unpreconditioned resid norm 3.599279687857e+00 true resid norm 1.170294925221e+02 ||r(i)||/||b|| 4.246796366369e-01
>>>>>>>>>> 10 KSP unpreconditioned resid norm 3.597803724329e+00 true resid norm 1.141700721441e+02 ||r(i)||/||b|| 4.143032983230e-01
>>>>>>>>>> 11 KSP unpreconditioned resid norm 3.596737615999e+00 true resid norm 1.148307006855e+02 ||r(i)||/||b|| 4.167006041890e-01
>>>>>>>>>> 12 KSP unpreconditioned resid norm 3.595710841264e+00 true resid norm 1.156257644592e+02 ||r(i)||/||b|| 4.195857520885e-01
>>>>>>>>>> 13 KSP unpreconditioned resid norm 3.586196423514e+00 true resid norm 1.161164780969e+02 ||r(i)||/||b|| 4.213664663754e-01
>>>>>>>>>> 14 KSP unpreconditioned resid norm 3.435157463728e+00 true resid norm 1.495255918191e+02 ||r(i)||/||b|| 5.426023187246e-01
>>>>>>>>>> 15 KSP unpreconditioned resid norm 3.380326213569e+00 true resid norm 1.691583628281e+02 ||r(i)||/||b|| 6.138462238173e-01
>>>>>>>>>> 16 KSP unpreconditioned resid norm 3.370184966977e+00 true resid norm 1.626583689771e+02 ||r(i)||/||b|| 5.902588787193e-01
>>>>>>>>>> 17 KSP unpreconditioned resid norm 3.362283769411e+00 true resid norm 1.734587903024e+02 ||r(i)||/||b|| 6.294517257967e-01
>>>>>>>>>> 18 KSP unpreconditioned resid norm 3.350650432378e+00 true resid norm 1.748843230757e+02 ||r(i)||/||b|| 6.346247358399e-01
>>>>>>>>>> 19 KSP unpreconditioned resid norm 3.349048765263e+00 true resid norm 1.744218218990e+02 ||r(i)||/||b|| 6.329463996579e-01
>>>>>>>>>> 20 KSP unpreconditioned resid norm 3.348988713038e+00 true resid norm 1.757544032962e+02 ||r(i)||/||b|| 6.377821053534e-01
>>>>>>>>>> 21 KSP unpreconditioned resid norm 3.348981727899e+00 true resid norm 1.736215517902e+02 ||r(i)||/||b|| 6.300423588755e-01
>>>>>>>>>> 22 KSP unpreconditioned resid norm 3.347278139858e+00 true resid norm 1.769503681529e+02 ||r(i)||/||b|| 6.421220534282e-01
>>>>>>>>>> 23 KSP unpreconditioned resid norm 3.219226414932e+00 true resid norm 1.769575879142e+02 ||r(i)||/||b|| 6.421482526842e-01
>>>>>>>>>> 24 KSP unpreconditioned resid norm 2.822590852350e+00 true resid norm 1.859375430783e+02 ||r(i)||/||b|| 6.747349452684e-01
>>>>>>>>>> 25 KSP unpreconditioned resid norm 2.309519054919e+00 true resid norm 2.168779675838e+02 ||r(i)||/||b|| 7.870123546051e-01
>>>>>>>>>> 26 KSP unpreconditioned resid norm 2.227139205444e+00 true resid norm 2.110752201996e+02 ||r(i)||/||b|| 7.659551954438e-01
>>>>>>>>>> 27 KSP unpreconditioned resid norm 1.071083203142e+00 true resid norm 2.654638231029e+02 ||r(i)||/||b|| 9.633219584739e-01
>>>>>>>>>> 28 KSP unpreconditioned resid norm 9.806030722653e-01 true resid norm 2.767219704819e+02 ||r(i)||/||b|| 1.004175813644e+00
>>>>>>>>>> 29 KSP unpreconditioned resid norm 9.664048337082e-01 true resid norm 2.735425797813e+02 ||r(i)||/||b|| 9.926383587826e-01
>>>>>>>>>> 30 KSP unpreconditioned resid norm 2.770004820147e+02 true resid norm 2.770004820147e+02 ||r(i)||/||b|| 1.005186483468e+00
>>>>>>>>>> 31 KSP unpreconditioned resid norm 1.140110593381e+02 true resid norm 2.945037649612e+02 ||r(i)||/||b|| 1.068702847433e+00
>>>>>>>>>> 32 KSP unpreconditioned resid norm 8.236226033426e+01 true resid norm 2.546580529874e+02 ||r(i)||/||b|| 9.241097015692e-01
>>>>>>>>>> 33 KSP unpreconditioned resid norm 3.898917963685e+01 true resid norm 2.745281955063e+02 ||r(i)||/||b|| 9.962149865105e-01
>>>>>>>>>> 34 KSP unpreconditioned resid norm 1.011205458583e+01 true resid norm 2.898261296072e+02 ||r(i)||/||b|| 1.051728523784e+00
>>>>>>>>>> 35 KSP unpreconditioned resid norm 8.616275790199e+00 true resid norm 3.018215534190e+02 ||r(i)||/||b|| 1.095257826663e+00
>>>>>>>>>> 36 KSP unpreconditioned resid norm 7.060253104363e+00 true resid norm 3.019791937262e+02 ||r(i)||/||b|| 1.095829875870e+00
>>>>>>>>>> 37 KSP unpreconditioned resid norm 6.570242477517e+00 true resid norm 2.929998255276e+02 ||r(i)||/||b|| 1.063245313281e+00
>>>>>>>>>> 38 KSP unpreconditioned resid norm 5.572927530093e+00 true resid norm 3.163520220055e+02 ||r(i)||/||b|| 1.147986365311e+00
>>>>>>>>>> 39 KSP unpreconditioned resid norm 5.293220961263e+00 true resid norm 2.951883280350e+02 ||r(i)||/||b|| 1.071187007546e+00
>>>>>>>>>> 40 KSP unpreconditioned resid norm 5.256809704195e+00 true resid norm 3.159367355815e+02 ||r(i)||/||b|| 1.146479363240e+00
>>>>>>>>>> 41 KSP unpreconditioned resid norm 5.190301236286e+00 true resid norm 3.075214639302e+02 ||r(i)||/||b|| 1.115941808731e+00
>>>>>>>>>> 42 KSP unpreconditioned resid norm 5.183649726411e+00 true resid norm 3.039070341991e+02 ||r(i)||/||b|| 1.102825673031e+00
>>>>>>>>>> 43 KSP unpreconditioned resid norm 5.118281808364e+00 true resid norm 2.958169749616e+02 ||r(i)||/||b|| 1.073468257705e+00
>>>>>>>>>> 44 KSP unpreconditioned resid norm 2.885097693426e+00 true resid norm 3.734759823549e+02 ||r(i)||/||b|| 1.355279263893e+00
>>>>>>>>>> 45 KSP unpreconditioned resid norm 1.035319192472e+00 true resid norm 4.155548703507e+02 ||r(i)||/||b|| 1.507976216422e+00
>>>>>>>>>> 46 KSP unpreconditioned resid norm 5.579863623125e-01 true resid norm 4.078820152216e+02 ||r(i)||/||b|| 1.480132762110e+00
>>>>>>>>>> 47 KSP unpreconditioned resid norm 5.120421018240e-01 true resid norm 4.092157058207e+02 ||r(i)||/||b|| 1.484972492906e+00
>>>>>>>>>> 48 KSP unpreconditioned resid norm 4.796787069079e-01 true resid norm 4.058092915545e+02 ||r(i)||/||b|| 1.472611209083e+00
>>>>>>>>>> 49 KSP unpreconditioned resid norm 4.744592839962e-01 true resid norm 4.173468194199e+02 ||r(i)||/||b|| 1.514478887357e+00
>>>>>>>>>> 50 KSP unpreconditioned resid norm 4.744564914486e-01 true resid norm 4.173567919777e+02 ||r(i)||/||b|| 1.514515076032e+00
>>>>>>>>>> 51 KSP unpreconditioned resid norm 4.744563271210e-01 true resid norm 4.171544676107e+02 ||r(i)||/||b|| 1.513780876158e+00
>>>>>>>>>> 52 KSP unpreconditioned resid norm 4.741552119642e-01 true resid norm 4.156339669040e+02 ||r(i)||/||b|| 1.508263244032e+00
>>>>>>>>>> 53 KSP unpreconditioned resid norm 4.260771566579e-01 true resid norm 4.963291201426e+02 ||r(i)||/||b|| 1.801091894462e+00
>>>>>>>>>> 54 KSP unpreconditioned resid norm 1.606767026862e-01 true resid norm 7.811691813982e+02 ||r(i)||/||b|| 2.834726844992e+00
>>>>>>>>>> 55 KSP unpreconditioned resid norm 1.438586734835e-01 true resid norm 8.207296621168e+02 ||r(i)||/||b|| 2.978284936331e+00
>>>>>>>>>> 56 KSP unpreconditioned resid norm 1.406571559968e-01 true resid norm 7.936946934103e+02 ||r(i)||/||b|| 2.880179745585e+00
>>>>>>>>>> 57 KSP unpreconditioned resid norm 1.314866726817e-01 true resid norm 7.959475933917e+02 ||r(i)||/||b|| 2.888355127062e+00
>>>>>>>>>> 58 KSP unpreconditioned resid norm 1.294094769063e-01 true resid norm 7.980559340873e+02 ||r(i)||/||b|| 2.896005928080e+00
>>>>>>>>>> 59 KSP unpreconditioned resid norm 1.157036938799e-01 true resid norm 8.192900430430e+02 ||r(i)||/||b|| 2.973060809557e+00
>>>>>>>>>> 60 KSP unpreconditioned resid norm 8.241203625923e+02 true resid norm 8.241203625923e+02 ||r(i)||/||b|| 2.990589197546e+00
>>>>>>>>>> 61 KSP unpreconditioned resid norm 5.762204692795e+02 true resid norm 1.139580328541e+03 ||r(i)||/||b|| 4.135338446864e+00
>>>>>>>>>> 62 KSP unpreconditioned resid norm 3.985894224287e+02 true resid norm 1.032497263075e+03 ||r(i)||/||b|| 3.746752660906e+00
>>>>>>>>>> 63 KSP unpreconditioned resid norm 3.715639719604e+02 true resid norm 1.129061045316e+03 ||r(i)||/||b|| 4.097165800964e+00
>>>>>>>>>> 64 KSP unpreconditioned resid norm 3.517510024064e+02 true resid norm 1.105272485190e+03 ||r(i)||/||b|| 4.010841261289e+00
>>>>>>>>>> 65 KSP unpreconditioned resid norm 3.505256741334e+02 true resid norm 1.119116680546e+03 ||r(i)||/||b|| 4.061079433966e+00
>>>>>>>>>> 66 KSP unpreconditioned resid norm 3.245282609350e+02 true resid norm 1.278512954302e+03 ||r(i)||/||b|| 4.639500737528e+00
>>>>>>>>>> 67 KSP unpreconditioned resid norm 3.185546922426e+02 true resid norm 1.316116102154e+03 ||r(i)||/||b|| 4.775956008945e+00
>>>>>>>>>> 68 KSP unpreconditioned resid norm 2.935614497897e+02 true resid norm 1.613148889104e+03 ||r(i)||/||b|| 5.853836236510e+00
>>>>>>>>>> 69 KSP unpreconditioned resid norm 2.916950130807e+02 true resid norm 1.640693935843e+03 ||r(i)||/||b|| 5.953792411558e+00
>>>>>>>>>> 70 KSP unpreconditioned resid norm 5.338203474651e+01 true resid norm 6.786538543779e+03 ||r(i)||/||b|| 2.462716585950e+01
>>>>>>>>>> 71 KSP unpreconditioned resid norm 3.842451984147e+01 true resid norm 7.031743784093e+03 ||r(i)||/||b|| 2.551697295098e+01
>>>>>>>>>> 72 KSP unpreconditioned resid norm 2.456635578348e+01 true resid norm 7.011081451208e+03 ||r(i)||/||b|| 2.544199294523e+01
>>>>>>>>>> 73 KSP unpreconditioned resid norm 1.197940743903e+01 true resid norm 7.057459572500e+03 ||r(i)||/||b|| 2.561029106628e+01
>>>>>>>>>> 74 KSP unpreconditioned resid norm 1.094051775599e+01 true resid norm 7.146713226914e+03 ||r(i)||/||b|| 2.593417702620e+01
>>>>>>>>>> 75 KSP unpreconditioned resid norm 1.086455030374e+01 true resid norm 7.120952027582e+03 ||r(i)||/||b|| 2.584069412257e+01
>>>>>>>>>> 76 KSP unpreconditioned resid norm 1.079928058658e+01 true resid norm 7.139110458055e+03 ||r(i)||/||b|| 2.590658790275e+01
>>>>>>>>>> 77 KSP unpreconditioned resid norm 1.075256031213e+01 true resid norm 7.144313896651e+03 ||r(i)||/||b|| 2.592547027475e+01
>>>>>>>>>> 78 KSP unpreconditioned resid norm 1.066313589844e+01 true resid norm 7.167082414494e+03 ||r(i)||/||b|| 2.600809325872e+01
>>>>>>>>>> 79 KSP unpreconditioned resid norm 1.059506075324e+01 true resid norm 7.090827855170e+03 ||r(i)||/||b|| 2.573137874985e+01
>>>>>>>>>> 80 KSP unpreconditioned resid norm 1.047508685484e+01 true resid norm 7.075992307465e+03 ||r(i)||/||b|| 2.567754313224e+01
>>>>>>>>>> 81 KSP unpreconditioned resid norm 1.043458037171e+01 true resid norm 7.088228807561e+03 ||r(i)||/||b|| 2.572194725895e+01
>>>>>>>>>> 82 KSP unpreconditioned resid norm 1.039892140666e+01 true resid norm 7.078316885367e+03 ||r(i)||/||b|| 2.568597862041e+01
>>>>>>>>>> 83 KSP unpreconditioned resid norm 1.037939199767e+01 true resid norm 7.066205554283e+03 ||r(i)||/||b|| 2.564202871023e+01
>>>>>>>>>> 84 KSP unpreconditioned resid norm 9.984750042542e+00 true resid norm 7.006338412442e+03 ||r(i)||/||b|| 2.542478128400e+01
>>>>>>>>>> 85 KSP unpreconditioned resid norm 9.883677466536e+00 true resid norm 7.058028159441e+03 ||r(i)||/||b|| 2.561235436922e+01
>>>>>>>>>> 86 KSP unpreconditioned resid norm 6.619460053034e+00 true resid norm 7.044190785882e+03 ||r(i)||/||b|| 2.556214095165e+01
>>>>>>>>>> 87 KSP unpreconditioned resid norm 6.310659447939e+00 true resid norm 6.988896164708e+03 ||r(i)||/||b|| 2.536148640619e+01
>>>>>>>>>> 88 KSP unpreconditioned resid norm 5.965222301448e+00 true resid norm 6.977241230228e+03 ||r(i)||/||b|| 2.531919267977e+01
>>>>>>>>>> 89 KSP unpreconditioned resid norm 5.937863010270e+00 true resid norm 6.984983705932e+03 ||r(i)||/||b|| 2.534728877501e+01
>>>>>>>>>> 90 KSP unpreconditioned resid norm 7.016500581832e+03 true resid norm 7.016500581832e+03 ||r(i)||/||b|| 2.546165802602e+01
>>>>>>>>>> 91 KSP unpreconditioned resid norm 5.093503615041e+03 true resid norm 6.330240058388e+03 ||r(i)||/||b|| 2.297133816344e+01
>>>>>>>>>> 92 KSP unpreconditioned resid norm 4.619237287853e+03 true resid norm 8.431713231372e+03 ||r(i)||/||b|| 3.059721813842e+01
>>>>>>>>>> 93 KSP unpreconditioned resid norm 4.597871280643e+03 true resid norm 8.292265533928e+03 ||r(i)||/||b|| 3.009118674236e+01
>>>>>>>>>> 94 KSP unpreconditioned resid norm 3.891645464749e+03 true resid norm 1.502202111405e+04 ||r(i)||/||b|| 5.451229712086e+01
>>>>>>>>>> 95 KSP unpreconditioned resid norm 3.200596462200e+03 true resid norm 2.668959459509e+04 ||r(i)||/||b|| 9.685188827501e+01
>>>>>>>>>> 96 KSP unpreconditioned resid norm 3.133449271747e+03 true resid norm 2.745239308481e+04 ||r(i)||/||b|| 9.961995108089e+01
>>>>>>>>>> 97 KSP unpreconditioned resid norm 3.132571481382e+03 true resid norm 2.746198607073e+04 ||r(i)||/||b|| 9.965476235528e+01
>>>>>>>>>> 98 KSP unpreconditioned resid norm 2.765126329282e+03 true resid norm 5.866172400570e+04 ||r(i)||/||b|| 2.128731749438e+02
>>>>>>>>>> 99 KSP unpreconditioned resid norm 2.727323406805e+03 true resid norm 6.346138079041e+04 ||r(i)||/||b|| 2.302902931026e+02
>>>>>>>>>> 100 KSP unpreconditioned resid norm 2.121814231223e+03 true resid norm 1.195380938846e+05 ||r(i)||/||b|| 4.337829138718e+02
>>>>>>>>>> 101 KSP unpreconditioned resid norm 2.009360588857e+03 true resid norm 1.304494792115e+05 ||r(i)||/||b|| 4.733784299761e+02
>>>>>>>>>> 102 KSP unpreconditioned resid norm 2.005896554452e+03 true resid norm 1.320230537653e+05 ||r(i)||/||b|| 4.790886578455e+02
>>>>>>>>>> 103 KSP unpreconditioned resid norm 1.956012050906e+03 true resid norm 1.367295899920e+05 ||r(i)||/||b|| 4.961678577250e+02
>>>>>>>>>> 104 KSP unpreconditioned resid norm 1.917942474919e+03 true resid norm 1.386791785519e+05 ||r(i)||/||b|| 5.032425749043e+02
>>>>>>>>>> 105 KSP unpreconditioned resid norm 1.891011896845e+03 true resid norm 1.430038847663e+05 ||r(i)||/||b|| 5.189361802006e+02
>>>>>>>>>> 106 KSP unpreconditioned resid norm 1.821613376258e+03 true resid norm 1.434421314555e+05 ||r(i)||/||b|| 5.205265010736e+02
>>>>>>>>>> 107 KSP unpreconditioned resid norm 1.495077331141e+03 true resid norm 1.694987590930e+05 ||r(i)||/||b|| 6.150814625505e+02
>>>>>>>>>> 108 KSP unpreconditioned resid norm 1.462526968856e+03 true resid norm 1.635061661924e+05 ||r(i)||/||b|| 5.933353870895e+02
>>>>>>>>>> 109 KSP unpreconditioned resid norm 1.457563548662e+03 true resid norm 1.638719165345e+05 ||r(i)||/||b|| 5.946626313513e+02
>>>>>>>>>> 110 KSP unpreconditioned resid norm 1.454261407177e+03 true resid norm 1.647934153621e+05 ||r(i)||/||b|| 5.980065900305e+02
>>>>>>>>>> 111 KSP unpreconditioned resid norm 1.453528774889e+03 true resid norm 1.628660507161e+05 ||r(i)||/||b|| 5.910125195626e+02
>>>>>>>>>> 112 KSP unpreconditioned resid norm 1.443038716768e+03 true resid norm 1.583680545044e+05 ||r(i)||/||b|| 5.746900750607e+02
>>>>>>>>>> 113 KSP unpreconditioned resid norm 1.437764618649e+03 true resid norm 1.608017231907e+05 ||r(i)||/||b|| 5.835214346703e+02
>>>>>>>>>> 114 KSP unpreconditioned resid norm 1.434261116035e+03 true resid norm 1.599123455153e+05 ||r(i)||/||b|| 5.802940380554e+02
>>>>>>>>>> 115 KSP unpreconditioned resid norm 1.424002324402e+03 true resid norm 1.579975804284e+05 ||r(i)||/||b|| 5.733456891919e+02
>>>>>>>>>> 116 KSP unpreconditioned resid norm 1.357749753091e+03 true resid norm 1.551082296226e+05 ||r(i)||/||b|| 5.628607385707e+02
>>>>>>>>>> 117 KSP unpreconditioned resid norm 1.198269521762e+03 true resid norm 1.545809594035e+05 ||r(i)||/||b|| 5.609473668193e+02
>>>>>>>>>> 118 KSP unpreconditioned resid norm 1.130943883259e+03 true resid norm 1.572138269970e+05 ||r(i)||/||b|| 5.705015845541e+02
>>>>>>>>>> 119 KSP unpreconditioned resid norm 1.125785379845e+03 true resid norm 1.547962006264e+05 ||r(i)||/||b|| 5.617284397127e+02
>>>>>>>>>> 120 KSP unpreconditioned resid norm 1.543427251551e+05 true resid norm 1.543427251551e+05 ||r(i)||/||b|| 5.600828562430e+02
>>>>>>>>>> 121 KSP unpreconditioned resid norm 9.687641861615e+03 true resid norm 1.312462482586e+05 ||r(i)||/||b|| 4.762697660157e+02
>>>>>>>>>> 122 KSP unpreconditioned resid norm 4.306237612170e+03 true resid norm 1.216805191948e+05 ||r(i)||/||b|| 4.415574020171e+02
>>>>>>>>>> 123 KSP unpreconditioned resid norm 3.475568157040e+03 true resid norm 1.229750886113e+05 ||r(i)||/||b|| 4.462551688584e+02
>>>>>>>>>> 124 KSP unpreconditioned resid norm 2.129292353503e+03 true resid norm 1.198465289785e+05 ||r(i)||/||b|| 4.349021710846e+02
>>>>>>>>>> 125 KSP unpreconditioned resid norm 1.870804830415e+03 true resid norm 1.184972385737e+05 ||r(i)||/||b|| 4.300058313119e+02
>>>>>>>>>> 126 KSP unpreconditioned resid norm 1.619045998374e+03 true resid norm 1.223524555012e+05 ||r(i)||/||b|| 4.439957417919e+02
>>>>>>>>>> 127 KSP unpreconditioned resid norm 1.376987056507e+03 true resid norm 1.178021963733e+05 ||r(i)||/||b|| 4.274836442738e+02
>>>>>>>>>> 128 KSP unpreconditioned resid norm 1.240725581528e+03 true resid norm 1.203158159107e+05 ||r(i)||/||b|| 4.366051315910e+02
>>>>>>>>>> 129 KSP unpreconditioned resid norm 1.149407701476e+03 true resid norm 1.195020992146e+05 ||r(i)||/||b|| 4.336522954863e+02
>>>>>>>>>> 130 KSP unpreconditioned resid norm 2.316334545969e+02 true resid norm 1.223507792260e+05 ||r(i)||/||b|| 4.439896588813e+02
>>>>>>>>>> 131 KSP unpreconditioned resid norm 8.374611050353e+01 true resid norm 1.201361156891e+05 ||r(i)||/||b|| 4.359530307985e+02
>>>>>>>>>> 132 KSP unpreconditioned resid norm 3.643538363030e+01 true resid norm 1.216626686116e+05 ||r(i)||/||b|| 4.414926253613e+02
>>>>>>>>>> 133 KSP unpreconditioned resid norm 2.717335002882e+01 true resid norm 1.209934243595e+05 ||r(i)||/||b|| 4.390640545824e+02
>>>>>>>>>> 134 KSP unpreconditioned resid norm 1.912772699959e+01 true resid norm 1.208873535853e+05 ||r(i)||/||b|| 4.386791422251e+02
>>>>>>>>>> 135 KSP unpreconditioned resid norm 1.524932868540e+00 true resid norm 1.230433708113e+05 ||r(i)||/||b|| 4.465029530645e+02
>>>>>>>>>> 136 KSP unpreconditioned resid norm 7.592632091308e-01 true resid norm 1.229456514344e+05 ||r(i)||/||b|| 4.461483464725e+02
>>>>>>>>>> 137 KSP unpreconditioned resid norm 6.118389339705e-01 true resid norm 1.228512430071e+05 ||r(i)||/||b|| 4.458057547402e+02
>>>>>>>>>> 138 KSP unpreconditioned resid norm 4.797543151006e-01 true resid norm 1.229234787855e+05 ||r(i)||/||b|| 4.460678857932e+02
>>>>>>>>>> 139 KSP unpreconditioned resid norm 1.399262290600e-01 true resid norm 1.229194801786e+05 ||r(i)||/||b|| 4.460533755456e+02
>>>>>>>>>> 140 KSP unpreconditioned resid norm 1.138136530735e-01 true resid norm 1.229195079694e+05 ||r(i)||/||b|| 4.460534763934e+02
>>>>>>>>>> 141 KSP unpreconditioned resid norm 9.664519186549e-02 true resid norm 1.229194652988e+05 ||r(i)||/||b|| 4.460533215493e+02
>>>>>>>>>> 142 KSP unpreconditioned resid norm 8.202341866260e-02 true resid norm 1.229193617253e+05 ||r(i)||/||b|| 4.460529456992e+02
>>>>>>>>>> 143 KSP unpreconditioned resid norm 5.682825851560e-02 true resid norm 1.228841671660e+05 ||r(i)||/||b|| 4.459252307758e+02
>>>>>>>>>> 144 KSP unpreconditioned resid norm 5.092809135755e-02 true resid norm 1.228796781564e+05 ||r(i)||/||b|| 4.459089409424e+02
>>>>>>>>>> 145 KSP unpreconditioned resid norm 4.946174382551e-02 true resid norm 1.228841500560e+05 ||r(i)||/||b|| 4.459251686865e+02
>>>>>>>>>> 146 KSP unpreconditioned resid norm 4.818582229493e-02 true resid norm 1.228841540010e+05 ||r(i)||/||b|| 4.459251830023e+02
>>>>>>>>>> 147 KSP unpreconditioned resid norm 4.804260791992e-02 true resid norm 1.228796714990e+05 ||r(i)||/||b|| 4.459089167835e+02
>>>>>>>>>> 148 KSP unpreconditioned resid norm 4.798036051442e-02 true resid norm 1.228796718963e+05 ||r(i)||/||b|| 4.459089182254e+02
>>>>>>>>>> 149 KSP unpreconditioned resid norm 4.789821323052e-02 true resid norm 1.228796773320e+05 ||r(i)||/||b|| 4.459089379507e+02
>>>>>>>>>> 150 KSP unpreconditioned resid norm 1.229170818997e+05 true resid norm 1.229170818997e+05 ||r(i)||/||b|| 4.460446726091e+02
>>>>>>>>>> 151 KSP unpreconditioned resid norm 1.504356443355e+04 true resid norm 1.093690560218e+05 ||r(i)||/||b|| 3.968812473652e+02
>>>>>>>>>> 152 KSP unpreconditioned resid norm 1.244195672870e+04 true resid norm 1.113856786937e+05 ||r(i)||/||b|| 4.041992196565e+02
>>>>>>>>>> 153 KSP unpreconditioned resid norm 1.225340186741e+04 true resid norm 1.167038059368e+05 ||r(i)||/||b|| 4.234977767679e+02
>>>>>>>>>> 154 KSP unpreconditioned resid norm 1.150472317241e+04 true resid norm 1.129983837286e+05 ||r(i)||/||b|| 4.100514452234e+02
>>>>>>>>>> 155 KSP unpreconditioned resid norm 4.442606181918e+03 true resid norm 1.317599156335e+05 ||r(i)||/||b|| 4.781337754155e+02
>>>>>>>>>> 156 KSP unpreconditioned resid norm 3.958747173080e+03 true resid norm 1.296136243806e+05 ||r(i)||/||b|| 4.703452584377e+02
>>>>>>>>>> 157 KSP unpreconditioned resid norm 3.611802792634e+03 true resid norm 1.294308648863e+05 ||r(i)||/||b|| 4.696820560778e+02
>>>>>>>>>> 158 KSP unpreconditioned resid norm 3.241244232472e+03 true resid norm 1.282759745135e+05 ||r(i)||/||b|| 4.654911601482e+02
>>>>>>>>>> 159 KSP unpreconditioned resid norm 2.768757317553e+03 true resid norm 1.270072256023e+05 ||r(i)||/||b|| 4.608870914215e+02
>>>>>>>>>> 160 KSP unpreconditioned resid norm 2.524612828404e+03 true resid norm 1.262142455939e+05 ||r(i)||/||b|| 4.580095051436e+02
>>>>>>>>>> 161 KSP unpreconditioned resid norm 1.597503632148e+03 true resid norm 1.242919647286e+05 ||r(i)||/||b|| 4.510338828299e+02
>>>>>>>>>> 162 KSP unpreconditioned resid norm 1.306334671172e+03 true resid norm 1.206147954536e+05 ||r(i)||/||b|| 4.376900762568e+02
>>>>>>>>>> 163 KSP unpreconditioned resid norm 1.002787050836e+03 true resid norm 1.211953380536e+05 ||r(i)||/||b|| 4.397967641960e+02
>>>>>>>>>> 164 KSP unpreconditioned resid norm 9.040547085337e+02 true resid norm 1.240638488124e+05 ||r(i)||/||b|| 4.502060899178e+02
>>>>>>>>>> 165 KSP unpreconditioned resid norm 8.266105040857e+02 true resid norm 1.217296905722e+05 ||r(i)||/||b|| 4.417358363780e+02
>>>>>>>>>> 166 KSP unpreconditioned resid norm 7.509799929433e+02 true resid norm 1.289718047507e+05 ||r(i)||/||b|| 4.680162068343e+02
>>>>>>>>>> 167 KSP unpreconditioned resid norm 6.343944680620e+02 true resid norm 1.221356169338e+05 ||r(i)||/||b|| 4.432088724138e+02
>>>>>>>>>> 168 KSP unpreconditioned resid norm 4.228561323994e+02 true resid norm 1.235824610762e+05 ||r(i)||/||b|| 4.484592177023e+02
>>>>>>>>>> 169 KSP unpreconditioned resid norm 7.102515984437e+01 true resid norm 1.260712062655e+05 ||r(i)||/||b|| 4.574904403444e+02
>>>>>>>>>> 170 KSP unpreconditioned resid norm 6.107553371402e+01 true resid norm 1.263309363140e+05 ||r(i)||/||b|| 4.584329554341e+02
>>>>>>>>>> 171 KSP unpreconditioned resid norm 5.628925898920e+01 true resid norm 1.265842262310e+05 ||r(i)||/||b|| 4.593521004084e+02
>>>>>>>>>> 172 KSP unpreconditioned resid norm 5.325842487623e+01 true resid norm 1.259756868046e+05 ||r(i)||/||b|| 4.571438168645e+02
>>>>>>>>>> 173 KSP unpreconditioned resid norm 4.313426816903e+01 true resid norm 1.256922861858e+05 ||r(i)||/||b|| 4.561154053999e+02
>>>>>>>>>> 174 KSP unpreconditioned resid norm 4.139644406511e+01 true resid norm 1.255866327936e+05 ||r(i)||/||b|| 4.557320076492e+02
>>>>>>>>>> 175 KSP unpreconditioned resid norm 4.059166270476e+01 true resid norm 1.257317172899e+05 ||r(i)||/||b|| 4.562584940061e+02
>>>>>>>>>> 176 KSP unpreconditioned resid norm 3.776505808733e+01 true resid norm 1.255329189055e+05 ||r(i)||/||b|| 4.555370893087e+02
>>>>>>>>>> 177 KSP unpreconditioned resid norm 3.752612693619e+01 true resid norm 1.256961684336e+05 ||r(i)||/||b|| 4.561294934007e+02
>>>>>>>>>> 178 KSP unpreconditioned resid norm 3.736705603634e+01 true resid norm 1.259762625890e+05 ||r(i)||/||b|| 4.571459062857e+02
>>>>>>>>>> 179 KSP unpreconditioned resid norm 3.732951008226e+01 true resid norm 1.261329420233e+05 ||r(i)||/||b|| 4.577144686526e+02
>>>>>>>>>> 180 KSP unpreconditioned resid norm 1.264843142503e+05 true resid norm 1.264843142503e+05 ||r(i)||/||b|| 4.589895372396e+02
>>>>>>>>>> 181 KSP unpreconditioned resid norm 8.819854345872e+04 true resid norm 7.421586684795e+04 ||r(i)||/||b|| 2.693164490971e+02
>>>>>>>>>> 182 KSP unpreconditioned resid norm 6.820284360474e+04 true resid norm 8.845099857884e+04 ||r(i)||/||b|| 3.209732617575e+02
>>>>>>>>>> 183 KSP unpreconditioned resid norm 6.649169133527e+04 true resid norm 9.695307886076e+04 ||r(i)||/||b|| 3.518258296613e+02
>>>>>>>>>> 184 KSP unpreconditioned resid norm 6.331239588602e+04 true resid norm 1.276316752783e+05 ||r(i)||/||b|| 4.631531104891e+02
>>>>>>>>>> 185 KSP unpreconditioned resid norm 6.252299705504e+04 true resid norm 1.347343905085e+05 ||r(i)||/||b|| 4.889276264515e+02
>>>>>>>>>> 186 KSP unpreconditioned resid norm 4.853904613639e+04 true resid norm 3.201242844765e+05 ||r(i)||/||b|| 1.161675248524e+03
>>>>>>>>>> 187 KSP unpreconditioned resid norm 4.219140436114e+04 true resid norm 4.404707106043e+05 ||r(i)||/||b|| 1.598391459258e+03
>>>>>>>>>> 188 KSP unpreconditioned resid norm 3.684315504592e+04 true resid norm 4.833177649269e+05 ||r(i)||/||b|| 1.753875953539e+03
>>>>>>>>>> 189 KSP unpreconditioned resid norm 3.534967798134e+04 true resid norm 5.018821663043e+05 ||r(i)||/||b|| 1.821242931396e+03
>>>>>>>>>> 190 KSP unpreconditioned resid norm 3.341243925885e+04 true resid norm 5.145529395860e+05 ||r(i)||/||b|| 1.867222959825e+03
>>>>>>>>>> 191 KSP unpreconditioned resid norm 3.297183308133e+04 true resid norm 5.078118375180e+05 ||r(i)||/||b|| 1.842760675019e+03
>>>>>>>>>> 192 KSP unpreconditioned resid norm 3.295138974490e+04 true resid norm 5.020778987445e+05 ||r(i)||/||b|| 1.821953210316e+03
>>>>>>>>>> 193 KSP unpreconditioned resid norm 3.293118332758e+04 true resid norm 5.057675768439e+05 ||r(i)||/||b|| 1.835342409233e+03
>>>>>>>>>> 194 KSP unpreconditioned resid norm 3.290271250575e+04 true resid norm 5.144670419966e+05 ||r(i)||/||b|| 1.866911252440e+03
>>>>>>>>>> 195 KSP unpreconditioned resid norm 3.286831121629e+04 true resid norm 5.172060400320e+05 ||r(i)||/||b|| 1.876850599056e+03
>>>>>>>>>> 196 KSP unpreconditioned resid norm 3.286666518120e+04 true resid norm 5.153103889545e+05 ||r(i)||/||b|| 1.869971611602e+03
>>>>>>>>>> 197 KSP unpreconditioned resid norm 3.285912278860e+04 true resid norm 5.129767922689e+05 ||r(i)||/||b|| 1.861503395846e+03
>>>>>>>>>> 198 KSP unpreconditioned resid norm 3.252356305725e+04 true resid norm 4.939344872669e+05 ||r(i)||/||b|| 1.792402189007e+03
>>>>>>>>>> 199 KSP unpreconditioned resid norm 3.251630298846e+04 true resid norm 4.923429687750e+05 ||r(i)||/||b|| 1.786626845713e+03
>>>>>>>>>> 200 KSP unpreconditioned resid norm 3.251525291518e+04 true resid norm 4.857429030576e+05 ||r(i)||/||b|| 1.762676357249e+03
>>>>>>>>>> 201 KSP unpreconditioned resid norm 3.222149134412e+04 true resid norm 6.182795797840e+05 ||r(i)||/||b|| 2.243628863324e+03
>>>>>>>>>> 202 KSP unpreconditioned resid norm 3.207404984650e+04 true resid norm 7.562732670579e+05 ||r(i)||/||b|| 2.744383909824e+03
>>>>>>>>>> 203 KSP unpreconditioned resid norm 3.207333995935e+04 true resid norm 7.589334030780e+05 ||r(i)||/||b|| 2.754037079927e+03
>>>>>>>>>> 204 KSP unpreconditioned resid norm 3.206223457945e+04 true resid norm 7.704831042122e+05 ||r(i)||/||b|| 2.795948932873e+03
>>>>>>>>>> 205 KSP unpreconditioned resid norm 3.204876215451e+04 true resid norm 7.555220845092e+05 ||r(i)||/||b|| 2.741657999243e+03
>>>>>>>>>> 206 KSP unpreconditioned resid norm 3.200091356228e+04 true resid norm 6.828836413819e+05 ||r(i)||/||b|| 2.478065745971e+03
>>>>>>>>>> 207 KSP unpreconditioned resid norm 3.086109249513e+04 true resid norm 1.425824444777e+06 ||r(i)||/||b|| 5.174068468269e+03
>>>>>>>>>> 208 KSP unpreconditioned resid norm 2.947470103154e+04 true resid norm 3.821305546130e+06 ||r(i)||/||b|| 1.386685198608e+04
>>>>>>>>>> 209 KSP unpreconditioned resid norm 2.923482810424e+04 true resid norm 4.118347357287e+06 ||r(i)||/||b|| 1.494476496091e+04
>>>>>>>>>> 210 KSP unpreconditioned resid norm 4.363951081769e+06 true resid norm 4.363951081769e+06 ||r(i)||/||b|| 1.583601808200e+04
>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINEAR_SOLVE iterations 24
>>>>>>>>>>
>>>>>>>>>> -gideon
>>>>>>>>>>
>>>>>>>>>>> On Sep 7, 2015, at 10:03 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is there any chance your Jacobian may have a null space? (For example with Neumann boundary conditions there is often a nullspace)
>>>>>>>>>>>
>>>>>>>>>>> If not run with with -ksp_type fgmres and send the same output.
>>>>>>>>>>>
>>>>>>>>>>> Something is screwy with the linear system convergence, could be several things including either a null space problem or that your Jacobian that you provide is terribly wrong.
>>>>>>>>>>>
>>>>>>>>>>> Barry
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Sep 7, 2015, at 8:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Got it, fixing that, and returning to the original question, this is what I now get, when I use those two flags:
>>>>>>>>>>>>
>>>>>>>>>>>> 0 SNES Function norm 1.132185384796e-08
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.846486104194e-09 true resid norm 1.132185384796e-08 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.813743504021e-14 true resid norm 4.325870388320e-13 ||r(i)||/||b|| 3.820814547168e-05
>>>>>>>>>>>> 1 SNES Function norm 2.177599365111e-12
>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.135087050041e+01 true resid norm 5.066222213176e+03 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.498184175046e-04 true resid norm 4.217739703330e-01 ||r(i)||/||b|| 8.325216553590e-05
>>>>>>>>>>>> 1 SNES Function norm 8.482593852817e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.667293100105e+02 true resid norm 8.482593852817e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.682642450862e-02 true resid norm 2.946982065167e+00 ||r(i)||/||b|| 3.474152029792e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.943933962588e-08 true resid norm 1.594643891705e-01 ||r(i)||/||b|| 1.879901265314e-04
>>>>>>>>>>>> 2 SNES Function norm 6.543140468549e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.538341367076e+02 true resid norm 6.543140468549e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.294478035133e-03 true resid norm 2.460794611648e-01 ||r(i)||/||b|| 3.760876942007e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.879002692918e-08 true resid norm 6.977344225129e-02 ||r(i)||/||b|| 1.066360145968e-04
>>>>>>>>>>>> 3 SNES Function norm 5.766430557220e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.362249760236e+02 true resid norm 5.766430557220e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.565721843210e-02 true resid norm 5.601235825427e-01 ||r(i)||/||b|| 9.713523417730e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.900043171759e-08 true resid norm 6.087965439352e-02 ||r(i)||/||b|| 1.055759777030e-04
>>>>>>>>>>>> 4 SNES Function norm 5.235211958260e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.533523981591e+02 true resid norm 5.235211958260e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.030969535942e-02 true resid norm 3.160110525368e-01 ||r(i)||/||b|| 6.036260901303e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.066614364159e-08 true resid norm 6.884372743978e-02 ||r(i)||/||b|| 1.315013183586e-04
>>>>>>>>>>>> 5 SNES Function norm 4.752913229649e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.848133067820e+02 true resid norm 4.752913229649e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.745259702307e-02 true resid norm 4.900437912538e-01 ||r(i)||/||b|| 1.031038791529e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.045643600156e-07 true resid norm 8.388613982820e-02 ||r(i)||/||b|| 1.764941537432e-04
>>>>>>>>>>>> 6 SNES Function norm 4.220255380391e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.549151165192e+02 true resid norm 4.220255380391e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.794436126782e-02 true resid norm 7.053097905720e-01 ||r(i)||/||b|| 1.671249076180e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.890094997586e-07 true resid norm 1.285407273503e-01 ||r(i)||/||b|| 3.045804477795e-04
>>>>>>>>>>>> 7 SNES Function norm 3.805408907074e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.225410707531e+02 true resid norm 3.805408907074e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.810631909992e-03 true resid norm 2.341273724704e-01 ||r(i)||/||b|| 6.152489211741e-04
>>>>>>>>>>>> 8 SNES Function norm 3.764619752339e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.256830807263e+03 true resid norm 3.764619752339e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.034356591467e+00 true resid norm 1.404676339819e+01 ||r(i)||/||b|| 3.731256892403e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.014544916729e-05 true resid norm 3.741004783833e+00 ||r(i)||/||b|| 9.937271304780e-03
>>>>>>>>>>>> 9 SNES Function norm 3.761182227091e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.988264999623e+03 true resid norm 3.761182227091e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.115167379340e-01 true resid norm 6.409965438916e+00 ||r(i)||/||b|| 1.704242190858e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.515961219888e-06 true resid norm 2.115087826060e+00 ||r(i)||/||b|| 5.623465438140e-03
>>>>>>>>>>>> 10 SNES Function norm 3.740017190063e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.142561025658e+03 true resid norm 3.740017190063e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.746046512685e-02 true resid norm 7.402316700861e-01 ||r(i)||/||b|| 1.979219967365e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.588755510089e-06 true resid norm 6.682186870202e-01 ||r(i)||/||b|| 1.786672769301e-03
>>>>>>>>>>>> 11 SNES Function norm 3.725903477238e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.435759974780e+03 true resid norm 3.725903477238e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 7.389097536786e-01 true resid norm 9.211825929813e+00 ||r(i)||/||b|| 2.472373744003e-02
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.785754916084e-06 true resid norm 8.219867808452e-01 ||r(i)||/||b|| 2.206140835013e-03
>>>>>>>>>>>> 12 SNES Function norm 3.716162097231e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.010267719047e+02 true resid norm 3.716162097231e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.404383077173e-01 true resid norm 2.467625408789e+00 ||r(i)||/||b|| 6.640252346978e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.431973894553e-07 true resid norm 2.411034602455e-01 ||r(i)||/||b|| 6.487969414067e-04
>>>>>>>>>>>> 13 SNES Function norm 3.674168632847e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.697546464095e+02 true resid norm 3.674168632847e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.100811362173e-01 true resid norm 1.260981844274e+00 ||r(i)||/||b|| 3.432019513205e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.921252511737e-07 true resid norm 1.549846956307e-01 ||r(i)||/||b|| 4.218224886174e-04
>>>>>>>>>>>> 14 SNES Function norm 3.532395445266e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.455211776575e+01 true resid norm 3.532395445266e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.681410748806e-02 true resid norm 7.130040117764e-02 ||r(i)||/||b|| 2.018471665543e-04
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.060849362790e-07 true resid norm 2.654058947496e-02 ||r(i)||/||b|| 7.513481966050e-05
>>>>>>>>>>>> 15 SNES Function norm 3.182438872366e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.515566740696e+02 true resid norm 3.182438872366e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.959786024091e-02 true resid norm 5.035985128637e-01 ||r(i)||/||b|| 1.582429492163e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.945498964959e-07 true resid norm 8.727232650514e-02 ||r(i)||/||b|| 2.742309593531e-04
>>>>>>>>>>>> 16 SNES Function norm 3.091759892779e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 7.834551377727e+01 true resid norm 3.091759892779e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.992168699741e-02 true resid norm 5.040095342951e-01 ||r(i)||/||b|| 1.630170361781e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 3.192091055088e-07 true resid norm 4.561900072730e-02 ||r(i)||/||b|| 1.475502701029e-04
>>>>>>>>>>>> 17 SNES Function norm 2.987839504359e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.384044359119e+02 true resid norm 2.987839504359e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.206851078700e-01 true resid norm 1.939469657735e+00 ||r(i)||/||b|| 6.491210973364e-03
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.074373330828e-06 true resid norm 5.530732277225e-01 ||r(i)||/||b|| 1.851080778990e-03
>>>>>>>>>>>> 18 SNES Function norm 2.987073622777e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 9.576546391157e+03 true resid norm 2.987073622777e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.670797073017e+00 true resid norm 3.409551583039e+01 ||r(i)||/||b|| 1.141435402542e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.278475684614e-05 true resid norm 6.179819087650e+00 ||r(i)||/||b|| 2.068853958110e-02
>>>>>>>>>>>> 19 SNES Function norm 2.987067936734e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.381986890301e+04 true resid norm 2.987067936734e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.870471171439e+01 true resid norm 2.593775594691e+02 ||r(i)||/||b|| 8.683349858880e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.628164409392e-04 true resid norm 2.546845295549e+01 ||r(i)||/||b|| 8.526238269403e-02
>>>>>>>>>>>> 20 SNES Function norm 2.987067502910e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.191539882357e+04 true resid norm 2.987067502910e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.528735548091e+01 true resid norm 1.390234679490e+02 ||r(i)||/||b|| 4.654178983689e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.890475574699e-05 true resid norm 1.018047198868e+01 ||r(i)||/||b|| 3.408182767468e-02
>>>>>>>>>>>> 21 SNES Function norm 2.987064584431e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.076322725615e+05 true resid norm 2.987064584431e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.990965035897e+01 true resid norm 1.933504170956e+02 ||r(i)||/||b|| 6.472923890006e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941661213523e-05 true resid norm 7.058455368301e+01 ||r(i)||/||b|| 2.363007283167e-01
>>>>>>>>>>>> 22 SNES Function norm 2.987064525262e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.312500251239e+06 true resid norm 2.987064525262e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.194187039215e+01 true resid norm 2.898773744566e+02 ||r(i)||/||b|| 9.704422921065e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.998846844269e-04 true resid norm 2.698070936922e+01 ||r(i)||/||b|| 9.032516419058e-02
>>>>>>>>>>>> 23 SNES Function norm 2.987064121622e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022854344216e+05 true resid norm 2.987064121622e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.602644501358e+00 true resid norm 8.634709464365e+01 ||r(i)||/||b|| 2.890701073962e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.288860381060e-05 true resid norm 4.471246585837e+01 ||r(i)||/||b|| 1.496869971244e-01
>>>>>>>>>>>> 24 SNES Function norm 2.987063973426e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.876191744212e+05 true resid norm 2.987063973426e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.701586788162e+01 true resid norm 4.356767962106e+02 ||r(i)||/||b|| 1.458545247395e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.533781032398e-04 true resid norm 1.001779383020e+02 ||r(i)||/||b|| 3.353725905879e-01
>>>>>>>>>>>> 25 SNES Function norm 2.987063920553e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.373321357604e+05 true resid norm 2.987063920553e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.804687152206e+02 true resid norm 2.541993895057e+03 ||r(i)||/||b|| 8.510008364956e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.556287913942e-03 true resid norm 8.346997816018e+02 ||r(i)||/||b|| 2.794382054761e+00
>>>>>>>>>>>> 26 SNES Function norm 2.987063919786e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.292036277396e+05 true resid norm 2.987063919786e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.282180994978e+01 true resid norm 2.073562667335e+02 ||r(i)||/||b|| 6.941808823037e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.163542587614e-04 true resid norm 2.367571914844e+01 ||r(i)||/||b|| 7.926083868380e-02
>>>>>>>>>>>> 27 SNES Function norm 2.987063393145e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.447152171843e+05 true resid norm 2.987063393145e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.283727214797e+01 true resid norm 2.070985084809e+02 ||r(i)||/||b|| 6.933180894527e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 4.585761865240e-04 true resid norm 4.702442627021e+01 ||r(i)||/||b|| 1.574269443967e-01
>>>>>>>>>>>> 28 SNES Function norm 2.987063253277e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.785016507221e+05 true resid norm 2.987063253277e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.095374466008e+01 true resid norm 3.817886040589e+02 ||r(i)||/||b|| 1.278140339479e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 9.617782709610e-05 true resid norm 1.003348221878e+02 ||r(i)||/||b|| 3.358978825700e-01
>>>>>>>>>>>> 29 SNES Function norm 2.987063197153e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.669977278077e+05 true resid norm 2.987063197153e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.767902454807e+01 true resid norm 2.821167168096e+02 ||r(i)||/||b|| 9.444618281879e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.086718418040e-04 true resid norm 1.840415528667e+02 ||r(i)||/||b|| 6.161287549660e-01
>>>>>>>>>>>> 30 SNES Function norm 2.987063193089e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.136061088127e+05 true resid norm 2.987063193089e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.186362823355e+02 true resid norm 1.132640635624e+03 ||r(i)||/||b|| 3.791820133718e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.719259676319e-04 true resid norm 6.778160111686e+02 ||r(i)||/||b|| 2.269171983830e+00
>>>>>>>>>>>> 31 SNES Function norm 2.987063192061e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.691271219294e+05 true resid norm 2.987063192061e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.462251245099e+01 true resid norm 4.074982952445e+02 ||r(i)||/||b|| 1.364210493864e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.183786013104e-04 true resid norm 7.539314959670e+01 ||r(i)||/||b|| 2.523989107331e-01
>>>>>>>>>>>> 32 SNES Function norm 2.987063094492e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.124446369819e+05 true resid norm 2.987063094492e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 8.973907556421e+01 true resid norm 8.420449781516e+02 ||r(i)||/||b|| 2.818972855660e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 7.374208523915e-04 true resid norm 2.422646074110e+02 ||r(i)||/||b|| 8.110461672459e-01
>>>>>>>>>>>> 33 SNES Function norm 2.987063089942e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.329193031679e+05 true resid norm 2.987063089942e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.183766435760e+01 true resid norm 2.005487212339e+02 ||r(i)||/||b|| 6.713909790161e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 6.941412108263e-05 true resid norm 2.615656494311e+01 ||r(i)||/||b|| 8.756616166289e-02
>>>>>>>>>>>> 34 SNES Function norm 2.987062660109e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.096972895505e+05 true resid norm 2.987062660109e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.733316558393e+01 true resid norm 3.375691658619e+02 ||r(i)||/||b|| 1.130104066346e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.068548659549e-04 true resid norm 3.231989034216e+01 ||r(i)||/||b|| 1.081995726898e-01
>>>>>>>>>>>> 35 SNES Function norm 2.987062107900e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.499649833972e+04 true resid norm 2.987062107900e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.096524448730e+01 true resid norm 2.899705639690e+02 ||r(i)||/||b|| 9.707550546139e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.801709551672e-04 true resid norm 7.425681993520e+01 ||r(i)||/||b|| 2.485948308166e-01
>>>>>>>>>>>> 36 SNES Function norm 2.987062055224e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 8.633609234208e+04 true resid norm 2.987062055224e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.229972441154e+01 true resid norm 1.120738026723e+02 ||r(i)||/||b|| 3.751974368137e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 5.664496361269e-05 true resid norm 3.120682958584e+01 ||r(i)||/||b|| 1.044733219763e-01
>>>>>>>>>>>> 37 SNES Function norm 2.987061774798e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.344743984989e+05 true resid norm 2.987061774798e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 2.372624802517e+01 true resid norm 2.243904583149e+02 ||r(i)||/||b|| 7.512079603043e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 2.527665232153e-04 true resid norm 7.229322099941e+01 ||r(i)||/||b|| 2.420211781669e-01
>>>>>>>>>>>> 38 SNES Function norm 2.987061715400e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.756130938265e+05 true resid norm 2.987061715400e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 4.629319937995e+01 true resid norm 4.389108492088e+02 ||r(i)||/||b|| 1.469373220332e+00
>>>>>>>>>>>> 2 KSP preconditioned resid norm 8.099878702234e-04 true resid norm 1.923430480373e+02 ||r(i)||/||b|| 6.439205693195e-01
>>>>>>>>>>>> 39 SNES Function norm 2.987061699634e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 2.326428261777e+05 true resid norm 2.987061699634e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 1.756089168490e+01 true resid norm 1.659759863394e+02 ||r(i)||/||b|| 5.556496752635e-01
>>>>>>>>>>>> 2 KSP preconditioned resid norm 1.956105557949e-04 true resid norm 6.351047187913e+01 ||r(i)||/||b|| 2.126185471392e-01
>>>>>>>>>>>> 40 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.133259484912e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.296568492777e+01 true resid norm 3.022811302762e+02 ||r(i)||/||b|| 1.011968173786e+00
>>>>>>>>>>>> 41 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 4.124224624453e+07 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.219934148122e+01 true resid norm 3.144341309207e+02 ||r(i)||/||b|| 1.052653643822e+00
>>>>>>>>>>>> 42 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.748707463105e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.291843121908e+01 true resid norm 3.293271430969e+02 ||r(i)||/||b|| 1.102512046562e+00
>>>>>>>>>>>> 43 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 1.022718794881e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314705152175e+01 true resid norm 3.015751234379e+02 ||r(i)||/||b|| 1.009604624165e+00
>>>>>>>>>>>> 44 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.380361366230e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.237699578864e+01 true resid norm 3.185658867057e+02 ||r(i)||/||b|| 1.066485818369e+00
>>>>>>>>>>>> 45 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 6.303672864381e+08 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.789746869236e+01 true resid norm 4.518602376852e+02 ||r(i)||/||b|| 1.512724856888e+00
>>>>>>>>>>>> 46 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 3.428971983722e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.314776104627e+01 true resid norm 2.999404779745e+02 ||r(i)||/||b|| 1.004132204557e+00
>>>>>>>>>>>> 47 SNES Function norm 2.987061630064e+02
>>>>>>>>>>>> 0 KSP preconditioned resid norm 5.189185292378e+09 true resid norm 2.987061630064e+02 ||r(i)||/||b|| 1.000000000000e+00
>>>>>>>>>>>> 1 KSP preconditioned resid norm 3.191097159135e+01 true resid norm 3.466943097246e+02 ||r(i)||/||b|| 1.160653353233e+00
>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 47
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -gideon
>>>>>>>>>>>>
>>>>>>>>>>>>> On Sep 7, 2015, at 9:39 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This indicates that somewhere in your ComputeJacobian you are setting matrix entries with the first Mat argument when you should always set them with the second matrix argument. For example if you have
>>>>>>>>>>>>>
>>>>>>>>>>>>> ComputeJacobian(SNES snes,Vec x, Mat J, Mat jpre,void *ctx)
>>>>>>>>>>>>>
>>>>>>>>>>>>> you should call all the MatSetValues() with jpre, no J. Then at the end of the function you should call MatAssemblyBegin/End() on jpre then on J if J is not == jpre see for example src/snes/examples/tutorials/ex1.c
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is a minor glitch we'll get past.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sep 7, 2015, at 8:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By the way, I tried using a different petsc installation, and now, rather than the segmentation fault, I get the following error:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>> [0]PETSC ERROR: No support for this operation for this object type
>>>>>>>>>>>>>> [0]PETSC ERROR: Mat type mffd
>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-darwin-c-debug named gs_air by gideon Mon Sep 7 21:32:18 2015
>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --download-mpich=yes --download-suitesparse=yes --download-superlu=yes --download-superlu_dist=yes --download-mumps=yes --download-sprng=yes --with-cxx=clang++ --with-cc=clang --with-fc=gfortran --download-metis=yes --download-parmetis=yes --download-scalapack=yes
>>>>>>>>>>>>>> [0]PETSC ERROR: #3892 MatSetValues() line 1116 in /opt/petsc-3.5.4/src/mat/interface/matrix.c
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:22 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hmm,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok you can try running it directly in the debugger since it is one process, type
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> gdb ./blowup_batch_refine
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> when the debugger comes up (if it does not cut and paste all output and send it)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> run -on_error_abort -snes_mf_operator and any other options you normally use
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:18 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Running with that flag gives me this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [0]PETSC ERROR: PETSC: Attaching gdb to ./blowup_batch_refine of pid 16111 on gs_air
>>>>>>>>>>>>>>>> Unable to start debugger: No such file or directory
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:11 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This should not happen. Run with a debug version of PETSc installed and the option -start_in_debugger noxterm Once the debugger starts up type cont and when it crashes type where or bt Send all output
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:09 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I’m getting an error with -snes_mf_operator,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 0 SNES Function norm 1.421454390131e-02
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: to get more information on the crash.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.5.3, unknown
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: ./blowup_batch_refine on a arch-macports named gs_air by gideon Mon Sep 7 21:08:19 2015
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options --prefix=/opt/local --prefix=/opt/local/lib/petsc --with-valgrind=0 --with-shared-libraries --with-debugging=0 --with-c2html-dir=/opt/local --with-x=0 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --with-hwloc-dir=/opt/local --with-suitesparse-dir=/opt/local --with-superlu-dir=/opt/local --with-metis-dir=/opt/local --with-parmetis-dir=/opt/local --with-scalapack-dir=/opt/local --with-mumps-dir=/opt/local --with-superlu_dist-dir=/opt/local CC=/opt/local/bin/mpicc-mpich-mp CXX=/opt/local/bin/mpicxx-mpich-mp FC=/opt/local/bin/mpif90-mpich-mp F77=/opt/local/bin/mpif90-mpich-mp F90=/opt/local/bin/mpif90-mpich-mp COPTFLAGS=-Os CXXOPTFLAGS=-Os FOPTFLAGS=-Os LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names" CPPFLAGS=-I/opt/local/include CFLAGS="-Os -arch x86_64" CXXFLAGS=-Os FFLAGS=-Os FCFLAGS=-Os F90FLAGS=-Os PETSC_ARCH=arch-macports --with-mpiexec=mpiexec-mpich-mp
>>>>>>>>>>>>>>>>>> [0]PETSC ERROR: #1 User provided function() line 0 in unknown file
>>>>>>>>>>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 9:01 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> My guess is the Jacobian is not correct (or correct "enough"), hence PETSc SNES is generating a poor descent direction. You can try
>>>>>>>>>>>>>>>>>>> -snes_mf_operator -ksp_monitor_true residual as additional arguments. What happens?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 7:49 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> No problem Matt, I don’t think we had previously discussed that output. Here is a case where things fail.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 4.027481756921e-09
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 1.760477878365e-12
>>>>>>>>>>>>>>>>>>>> Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 5.066222213176e+03
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.484697184230e+02
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 6.549559723294e+02
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 5.770723278153e+02
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 5.237702240594e+02
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 4.753909019848e+02
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 4.221784590755e+02
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 3.806525080483e+02
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 3.762054656019e+02
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 3.758975226873e+02
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 3.757032042706e+02
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 3.728798164234e+02
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 3.723078741075e+02
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 3.721848059825e+02
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 3.720227575629e+02
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 3.720051998555e+02
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 3.718945430587e+02
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 3.700412694044e+02
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 3.351964889461e+02
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 3.096016086233e+02
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 3.008410789787e+02
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 2.752316716557e+02
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 2.707658474165e+02
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 2.698436736049e+02
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 2.618233857172e+02
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 2.600121920634e+02
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 2.585046423168e+02
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 2.568551090220e+02
>>>>>>>>>>>>>>>>>>>> 28 SNES Function norm 2.556404537064e+02
>>>>>>>>>>>>>>>>>>>> 29 SNES Function norm 2.536353523683e+02
>>>>>>>>>>>>>>>>>>>> 30 SNES Function norm 2.533596070171e+02
>>>>>>>>>>>>>>>>>>>> 31 SNES Function norm 2.532324379596e+02
>>>>>>>>>>>>>>>>>>>> 32 SNES Function norm 2.531842335211e+02
>>>>>>>>>>>>>>>>>>>> 33 SNES Function norm 2.531684527520e+02
>>>>>>>>>>>>>>>>>>>> 34 SNES Function norm 2.531637604618e+02
>>>>>>>>>>>>>>>>>>>> 35 SNES Function norm 2.531624767821e+02
>>>>>>>>>>>>>>>>>>>> 36 SNES Function norm 2.531621359093e+02
>>>>>>>>>>>>>>>>>>>> 37 SNES Function norm 2.531620504925e+02
>>>>>>>>>>>>>>>>>>>> 38 SNES Function norm 2.531620350055e+02
>>>>>>>>>>>>>>>>>>>> 39 SNES Function norm 2.531620310522e+02
>>>>>>>>>>>>>>>>>>>> 40 SNES Function norm 2.531620300471e+02
>>>>>>>>>>>>>>>>>>>> 41 SNES Function norm 2.531620298084e+02
>>>>>>>>>>>>>>>>>>>> 42 SNES Function norm 2.531620297478e+02
>>>>>>>>>>>>>>>>>>>> 43 SNES Function norm 2.531620297324e+02
>>>>>>>>>>>>>>>>>>>> 44 SNES Function norm 2.531620297303e+02
>>>>>>>>>>>>>>>>>>>> 45 SNES Function norm 2.531620297302e+02
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 45
>>>>>>>>>>>>>>>>>>>> 0 SNES Function norm 9.636339304380e+03
>>>>>>>>>>>>>>>>>>>> 1 SNES Function norm 8.997731184634e+03
>>>>>>>>>>>>>>>>>>>> 2 SNES Function norm 8.120498349232e+03
>>>>>>>>>>>>>>>>>>>> 3 SNES Function norm 7.322379894820e+03
>>>>>>>>>>>>>>>>>>>> 4 SNES Function norm 6.599581599149e+03
>>>>>>>>>>>>>>>>>>>> 5 SNES Function norm 6.374872854688e+03
>>>>>>>>>>>>>>>>>>>> 6 SNES Function norm 6.372518007653e+03
>>>>>>>>>>>>>>>>>>>> 7 SNES Function norm 6.073996314301e+03
>>>>>>>>>>>>>>>>>>>> 8 SNES Function norm 5.635965277054e+03
>>>>>>>>>>>>>>>>>>>> 9 SNES Function norm 5.155389064046e+03
>>>>>>>>>>>>>>>>>>>> 10 SNES Function norm 5.080567902638e+03
>>>>>>>>>>>>>>>>>>>> 11 SNES Function norm 5.058878643969e+03
>>>>>>>>>>>>>>>>>>>> 12 SNES Function norm 5.058835649793e+03
>>>>>>>>>>>>>>>>>>>> 13 SNES Function norm 5.058491285707e+03
>>>>>>>>>>>>>>>>>>>> 14 SNES Function norm 5.057452865337e+03
>>>>>>>>>>>>>>>>>>>> 15 SNES Function norm 5.057226140688e+03
>>>>>>>>>>>>>>>>>>>> 16 SNES Function norm 5.056651272898e+03
>>>>>>>>>>>>>>>>>>>> 17 SNES Function norm 5.056575190057e+03
>>>>>>>>>>>>>>>>>>>> 18 SNES Function norm 5.056574632598e+03
>>>>>>>>>>>>>>>>>>>> 19 SNES Function norm 5.056574520229e+03
>>>>>>>>>>>>>>>>>>>> 20 SNES Function norm 5.056574492569e+03
>>>>>>>>>>>>>>>>>>>> 21 SNES Function norm 5.056574485124e+03
>>>>>>>>>>>>>>>>>>>> 22 SNES Function norm 5.056574483029e+03
>>>>>>>>>>>>>>>>>>>> 23 SNES Function norm 5.056574482427e+03
>>>>>>>>>>>>>>>>>>>> 24 SNES Function norm 5.056574482302e+03
>>>>>>>>>>>>>>>>>>>> 25 SNES Function norm 5.056574482287e+03
>>>>>>>>>>>>>>>>>>>> 26 SNES Function norm 5.056574482282e+03
>>>>>>>>>>>>>>>>>>>> 27 SNES Function norm 5.056574482281e+03
>>>>>>>>>>>>>>>>>>>> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 27
>>>>>>>>>>>>>>>>>>>> SNES Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: newtonls
>>>>>>>>>>>>>>>>>>>> maximum iterations=50, maximum function evaluations=10000
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-08, absolute=1e-50, solution=1e-08
>>>>>>>>>>>>>>>>>>>> total number of linear solver iterations=28
>>>>>>>>>>>>>>>>>>>> total number of function evaluations=323
>>>>>>>>>>>>>>>>>>>> total number of grid sequence refinements=2
>>>>>>>>>>>>>>>>>>>> SNESLineSearch Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: bt
>>>>>>>>>>>>>>>>>>>> interpolation: cubic
>>>>>>>>>>>>>>>>>>>> alpha=1.000000e-04
>>>>>>>>>>>>>>>>>>>> maxstep=1.000000e+08, minlambda=1.000000e-12
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
>>>>>>>>>>>>>>>>>>>> maximum iterations=40
>>>>>>>>>>>>>>>>>>>> KSP Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: gmres
>>>>>>>>>>>>>>>>>>>> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>>>>>>>>>>>>> GMRES: happy breakdown tolerance 1e-30
>>>>>>>>>>>>>>>>>>>> maximum iterations=10000, initial guess is zero
>>>>>>>>>>>>>>>>>>>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>>>>>>>>>>>>> left preconditioning
>>>>>>>>>>>>>>>>>>>> using PRECONDITIONED norm type for convergence test
>>>>>>>>>>>>>>>>>>>> PC Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: lu
>>>>>>>>>>>>>>>>>>>> LU: out-of-place factorization
>>>>>>>>>>>>>>>>>>>> tolerance for zero pivot 2.22045e-14
>>>>>>>>>>>>>>>>>>>> matrix ordering: nd
>>>>>>>>>>>>>>>>>>>> factor fill ratio given 0, needed 0
>>>>>>>>>>>>>>>>>>>> Factored matrix follows:
>>>>>>>>>>>>>>>>>>>> Mat Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>> package used to perform factorization: mumps
>>>>>>>>>>>>>>>>>>>> total: nonzeros=255801, allocated nonzeros=255801
>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =0
>>>>>>>>>>>>>>>>>>>> MUMPS run parameters:
>>>>>>>>>>>>>>>>>>>> SYM (matrix type): 0
>>>>>>>>>>>>>>>>>>>> PAR (host participation): 1
>>>>>>>>>>>>>>>>>>>> ICNTL(1) (output for error): 6
>>>>>>>>>>>>>>>>>>>> ICNTL(2) (output of diagnostic msg): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(3) (output for global info): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(4) (level of printing): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(5) (input mat struct): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(6) (matrix prescaling): 7
>>>>>>>>>>>>>>>>>>>> ICNTL(7) (sequentia matrix ordering):6
>>>>>>>>>>>>>>>>>>>> ICNTL(8) (scalling strategy): 77
>>>>>>>>>>>>>>>>>>>> ICNTL(10) (max num of refinements): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(11) (error analysis): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(12) (efficiency control): 1
>>>>>>>>>>>>>>>>>>>> ICNTL(13) (efficiency control): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(14) (percentage of estimated workspace increase): 20
>>>>>>>>>>>>>>>>>>>> ICNTL(18) (input mat struct): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(19) (Shur complement info): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(20) (rhs sparse pattern): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(21) (somumpstion struct): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(22) (in-core/out-of-core facility): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(23) (max size of memory can be allocated locally):0
>>>>>>>>>>>>>>>>>>>> ICNTL(24) (detection of null pivot rows): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(25) (computation of a null space basis): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(26) (Schur options for rhs or solution): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(27) (experimental parameter): -8
>>>>>>>>>>>>>>>>>>>> ICNTL(28) (use parallel or sequential ordering): 1
>>>>>>>>>>>>>>>>>>>> ICNTL(29) (parallel ordering): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(30) (user-specified set of entries in inv(A)): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(31) (factors is discarded in the solve phase): 0
>>>>>>>>>>>>>>>>>>>> ICNTL(33) (compute determinant): 0
>>>>>>>>>>>>>>>>>>>> CNTL(1) (relative pivoting threshold): 0.01
>>>>>>>>>>>>>>>>>>>> CNTL(2) (stopping criterion of refinement): 1.49012e-08
>>>>>>>>>>>>>>>>>>>> CNTL(3) (absomumpste pivoting threshold): 0
>>>>>>>>>>>>>>>>>>>> CNTL(4) (vamumpse of static pivoting): -1
>>>>>>>>>>>>>>>>>>>> CNTL(5) (fixation for null pivots): 0
>>>>>>>>>>>>>>>>>>>> RINFO(1) (local estimated flops for the elimination after analysis):
>>>>>>>>>>>>>>>>>>>> [0] 1.95838e+06
>>>>>>>>>>>>>>>>>>>> RINFO(2) (local estimated flops for the assembly after factorization):
>>>>>>>>>>>>>>>>>>>> [0] 143924
>>>>>>>>>>>>>>>>>>>> RINFO(3) (local estimated flops for the elimination after factorization):
>>>>>>>>>>>>>>>>>>>> [0] 1.95943e+06
>>>>>>>>>>>>>>>>>>>> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization):
>>>>>>>>>>>>>>>>>>>> [0] 7
>>>>>>>>>>>>>>>>>>>> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization):
>>>>>>>>>>>>>>>>>>>> [0] 7
>>>>>>>>>>>>>>>>>>>> INFO(23) (num of pivots eliminated on this processor after factorization):
>>>>>>>>>>>>>>>>>>>> [0] 15991
>>>>>>>>>>>>>>>>>>>> RINFOG(1) (global estimated flops for the elimination after analysis): 1.95838e+06
>>>>>>>>>>>>>>>>>>>> RINFOG(2) (global estimated flops for the assembly after factorization): 143924
>>>>>>>>>>>>>>>>>>>> RINFOG(3) (global estimated flops for the elimination after factorization): 1.95943e+06
>>>>>>>>>>>>>>>>>>>> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0,0)*(2^0)
>>>>>>>>>>>>>>>>>>>> INFOG(3) (estimated real workspace for factors on all processors after analysis): 255801
>>>>>>>>>>>>>>>>>>>> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 127874
>>>>>>>>>>>>>>>>>>>> INFOG(5) (estimated maximum front size in the complete tree): 11
>>>>>>>>>>>>>>>>>>>> INFOG(6) (number of nodes in the complete tree): 3996
>>>>>>>>>>>>>>>>>>>> INFOG(7) (ordering option effectively use after analysis): 6
>>>>>>>>>>>>>>>>>>>> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 86
>>>>>>>>>>>>>>>>>>>> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 255865
>>>>>>>>>>>>>>>>>>>> INFOG(10) (total integer space store the matrix factors after factorization): 127890
>>>>>>>>>>>>>>>>>>>> INFOG(11) (order of largest frontal matrix after factorization): 11
>>>>>>>>>>>>>>>>>>>> INFOG(12) (number of off-diagonal pivots): 19
>>>>>>>>>>>>>>>>>>>> INFOG(13) (number of delayed pivots after factorization): 8
>>>>>>>>>>>>>>>>>>>> INFOG(14) (number of memory compress after factorization): 0
>>>>>>>>>>>>>>>>>>>> INFOG(15) (number of steps of iterative refinement after solution): 0
>>>>>>>>>>>>>>>>>>>> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 7
>>>>>>>>>>>>>>>>>>>> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 7
>>>>>>>>>>>>>>>>>>>> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 7
>>>>>>>>>>>>>>>>>>>> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 7
>>>>>>>>>>>>>>>>>>>> INFOG(20) (estimated number of entries in the factors): 255801
>>>>>>>>>>>>>>>>>>>> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 7
>>>>>>>>>>>>>>>>>>>> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 7
>>>>>>>>>>>>>>>>>>>> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0
>>>>>>>>>>>>>>>>>>>> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1
>>>>>>>>>>>>>>>>>>>> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0
>>>>>>>>>>>>>>>>>>>> INFOG(28) (after factorization: number of null pivots encountered): 0
>>>>>>>>>>>>>>>>>>>> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 255865
>>>>>>>>>>>>>>>>>>>> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 5, 5
>>>>>>>>>>>>>>>>>>>> INFOG(32) (after analysis: type of analysis done): 1
>>>>>>>>>>>>>>>>>>>> INFOG(33) (value used for ICNTL(8)): 7
>>>>>>>>>>>>>>>>>>>> INFOG(34) (exponent of the determinant if determinant is requested): 0
>>>>>>>>>>>>>>>>>>>> linear system matrix = precond matrix:
>>>>>>>>>>>>>>>>>>>> Mat Object: 1 MPI processes
>>>>>>>>>>>>>>>>>>>> type: seqaij
>>>>>>>>>>>>>>>>>>>> rows=15991, cols=15991
>>>>>>>>>>>>>>>>>>>> total: nonzeros=223820, allocated nonzeros=431698
>>>>>>>>>>>>>>>>>>>> total number of mallocs used during MatSetValues calls =15991
>>>>>>>>>>>>>>>>>>>> using I-node routines: found 4000 nodes, limit used is 5
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sep 7, 2015, at 8:40 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2015 at 7:32 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> Barry,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I finally got a chance to really try using the grid sequencing within my code. I find that, in some cases, even if it can solve successfully on the coarsest mesh, the SNES fails, usually due to a line search failure, when it tries to compute along the grid sequence. Would you have any suggestions?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I apologize if I have asked before, but can you give me -snes_view for the solver? I could not find it in the email thread.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would suggest trying to fiddle with the line search, or precondition it with Richardson. It would be nice to see -snes_monitor
>>>>>>>>>>>>>>>>>>>>> for the runs that fail, and then we can break down the residual into fields and look at it again (if my custom residual monitor
>>>>>>>>>>>>>>>>>>>>> does not work we can write one easily). Seeing which part of the residual does not converge is key to designing the NASM
>>>>>>>>>>>>>>>>>>>>> for the problem. I have just seen the virtuoso of this, Xiao-Chuan Cai, present it. We need better monitoring in PETSc.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Matt
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 4:21 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:04 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yes, if i continue in this parameter on the coarse mesh, I can generally solve at all values. I do find that I need to do some amount of continuation to solve near the endpoint. The problem is that on the coarse mesh, things are not fully resolved at all the values along the continuation parameter, and I would like to do refinement.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> One subtlety is that I actually want the intermediate continuation solutions too. Currently, without doing any grid sequence, I compute each, write it to disk, and then go on to the next one. So I now need to go back an refine them. I was thinking that perhaps I could refine them on the fly, dump them to disk, and use the coarse solution as the starting guess at the next iteration, but that would seem to require resetting the snes back to the coarse grid.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The alternative would be to just script the mesh refinement in a post processing stage, where each value of the continuation is parameter is loaded on the coarse mesh, and refined. Perhaps that’s the most practical thing to do.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would do the following. Create your DM and create a SNES that will do the continuation
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> loop over continuation parameter
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snes,NULL,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> if (you decide you want to see the refined solution at this continuation point) {
>>>>>>>>>>>>>>>>>>>>>> SNESCreate(comm,&snesrefine);
>>>>>>>>>>>>>>>>>>>>>> SNESSetDM()
>>>>>>>>>>>>>>>>>>>>>> etc
>>>>>>>>>>>>>>>>>>>>>> SNESSetGridSequence(snesrefine,)
>>>>>>>>>>>>>>>>>>>>>> SNESSolve(snesrefine,0,Ucoarse);
>>>>>>>>>>>>>>>>>>>>>> SNESGetSolution(snesrefine,&Ufine);
>>>>>>>>>>>>>>>>>>>>>> VecView(Ufine or do whatever you want to do with the Ufine at that continuation point
>>>>>>>>>>>>>>>>>>>>>> SNESDestroy(snesrefine);
>>>>>>>>>>>>>>>>>>>>>> end if
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> end loop over continuation parameter.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Barry
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -gideon
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Aug 28, 2015, at 3:55 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 3. This problem is actually part of a continuation problem that roughly looks like this
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> for( continuation parameter p = 0 to 1){
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> solve with parameter p_i using solution from p_{i-1},
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> What I would like to do is to start the solver, for each value of parameter p_i on the coarse mesh, and then do grid sequencing on that. But it appears that after doing grid sequencing on the initial p_0 = 0, the SNES is set to use the finer mesh.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> So you are using continuation to give you a good enough initial guess on the coarse level to even get convergence on the coarse level? First I would check if you even need the continuation (or can you not even solve the coarse problem without it).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> If you do need the continuation then you will need to tweak how you do the grid sequencing. I think this will work:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Do not use -snes_grid_sequencing
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Run SNESSolve() as many times as you want with your continuation parameter. This will all happen on the coarse mesh.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Call SNESSetGridSequence()
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Then call SNESSolve() again and it will do one solve on the coarse level and then interpolate to the next level etc.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>>>>>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>
>>
>
More information about the petsc-users
mailing list