<div dir="ltr"><div>Barry, thank you for the tips. Besides the trust region method, I have also tested line search methods. -snes_linesearch_type cp worked robustly. Other line search types didn't converge, except for basic. I'll spend some more time to check if my Jacobian is wrong or if -snes_mf_operator has some problem for my problem.<br>
<br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 29, 2014 at 5:02 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=""><br>
On Apr 29, 2014, at 9:19 AM, Norihiro Watanabe <<a href="mailto:norihiro.w@gmail.com">norihiro.w@gmail.com</a>> wrote:<br>
<br>
> Hi Barry,<br>
><br>
> Is it possible that -snes_mf_operator makes convergence of linear solves slower if unknowns are poorly scaled for multiphysics problems? I gave up to check Jacobian for the large problem because it takes too long time. Instead I tested it with several different small size problems and noticed scaling of my unknowns makes a difference in FD Jacobian. My unknowns are two kinds: pressure (1e7) and temperature (1e3).<br>
<br>
</div> This is not really a huge difference in scaling. In fact it seems pretty minor to me.<br>
<div class=""><br>
> I scaled pressure by 1e-5 and got the following different results from -snes_type test<br>
<br>
</div> Having a good scaling is always a good idea.<br>
<div class="">><br>
> without scaling<br>
> Norm of matrix ratio 3.81371e-05 difference 6.42349e+06 (user-defined state)<br>
><br>
> with scaling<br>
> Norm of matrix ratio 8.69182e-09 difference 1463.98 (user-defined state)<br>
><br>
> which may suggest that a differentiate parameter h is not properly set if unknowns are poorly scaled. I also tested the scaling for the large problem with keeping fluid properties constant and got the following different convergence behaviours:<br>
><br>
> without -snes_mf_operator<br>
> 0 SNES Function norm 4.425457683773e+04<br>
> 0 KSP Residual norm 44254.6<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
> 1 KSP Residual norm 0.000168321<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
> 2 KSP Residual norm 8.18977e-06<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
> 3 KSP Residual norm 4.75882e-07<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 4 KSP Residual norm 4.06286e-08<br>
> Linear solve converged due to CONVERGED_RTOL iterations 4<br>
> 1 SNES Function norm 2.229156139237e+05<br>
<br>
</div> I don’t like this. Your SNES function norm has increased. I would avoid using a line search of basic and want to see some real decrease in the residual norm.<br>
<br>
Have you thought about doing some grid sequencing or some other continuation method to solve the nonlinear system?<br>
<br>
You could also try running with quad precision (./configure —with-precision=__float128 and gnu compilers)<br>
<br>
In my experience most failures of Newton do not come from “difficult physics” but rather from some error in the model (making it crazy), or the function or Jacobian evaluation.<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> with -snes_mf_operator<br>
> 0 SNES Function norm 4.425457683883e+04<br>
> 0 KSP Residual norm 44254.6<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
> 1 KSP Residual norm 5255.66<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 2 KSP Residual norm 1646.58<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 3 KSP Residual norm 1518.05<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> ...<br>
> 42 KSP Residual norm 0.656962<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 43 KSP Residual norm 0.462202<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
><br>
> with -snes_mf_operator and scaling of pressure by 1e-5<br>
> 0 SNES Function norm 4.425457683773e+04<br>
> 0 KSP Residual norm 44254.6<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
> 1 KSP Residual norm 1883.94<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 2 KSP Residual norm 893.88<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> ...<br>
> 42 KSP Residual norm 6.66081e-08<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 43 KSP Residual norm 2.17062e-08<br>
> Linear solve converged due to CONVERGED_RTOL iterations 43<br>
> 1 SNES Function norm 2.200867439822e+05<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Mon, Apr 28, 2014 at 5:59 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> It will take a very long time<br>
><br>
> On Apr 28, 2014, at 9:14 AM, Norihiro Watanabe <<a href="mailto:norihiro.w@gmail.com">norihiro.w@gmail.com</a>> wrote:<br>
><br>
> > I cannot surely say my Jacobian for this particular problem is correct, as I have not checked it. For a smaller problem, I've already checked its correctness using -snes_type test or -snes_compare_explicit (but linear solve and nonlinear solve with FD one need a few more iterations than with my Jacobian). To make it sure, now I started -snes_type test for the problem and will update you once it finished. By the way, I'm waiting the calculation for more than three hours now. Is it usual for a large problem (>1e6 dof) or is there something wrong?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Apr 28, 2014 at 6:34 AM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > I have run your code. I changed to use -snes_type newtonls and also -snes_mf_operator there is something wrong with your Jacobian:<br>
> ><br>
> > Without -snes_mf_operator<br>
> > 0 SNES Function norm 1.821611413735e+03<br>
> > 0 KSP Residual norm 1821.61<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 1 KSP Residual norm 0.000167024<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 2 KSP Residual norm 7.66595e-06<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 3 KSP Residual norm 4.4581e-07<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 4 KSP Residual norm 3.77537e-08<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 5 KSP Residual norm 2.20453e-09<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 6 KSP Residual norm 1.711e-10<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 6<br>
> ><br>
> > with -snes_mf_operator<br>
> ><br>
> > 0 SNES Function norm 1.821611413735e+03<br>
> > 0 KSP Residual norm 1821.61<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 1 KSP Residual norm 1796.39<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 2 KSP Residual norm 1786.2<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 3 KSP Residual norm 1741.11<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 4 KSP Residual norm 1733.92<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 5 KSP Residual norm 1726.57<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 6 KSP Residual norm 1725.35<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 7 KSP Residual norm 1723.89<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 8 KSP Residual norm 1715.41<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 9 KSP Residual norm 1713.72<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> > 10 KSP Residual norm 1702.84<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> ><br>
> > …<br>
> ><br>
> > This means your Jacobian is wrong. Your first order of business is to fix your Jacobian. I noticed in previous emails your discussion with Jed about switching to MatGetLocalSubMatrix() and using -snes_type test YOU NEED TO DO THIS. You will get no where with an incorrect Jacobian. You need to fix your Jacobian before you do anything else! No amount of other options or methods will help you with a wrong Jacobian! Once you have a correct Jacobian if you still have convergence problems let us know and we can make further suggestions.<br>
> ><br>
> > Barry<br>
> ><br>
> > On Apr 25, 2014, at 7:31 AM, Norihiro Watanabe <<a href="mailto:norihiro.w@gmail.com">norihiro.w@gmail.com</a>> wrote:<br>
> ><br>
> > > Hi,<br>
> > ><br>
> > > In my simulation, nonlinear solve with the trust regtion method got stagnent after linear solve (see output below). Is it possible that the method goes to inifite loop? Is there any parameter to avoid this situation?<br>
> > ><br>
> > > 0 SNES Function norm 1.828728087153e+03<br>
> > > 0 KSP Residual norm 91.2735<br>
> > > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > > Linear solve converged due to CONVERGED_RTOL iterations 3<br>
> > > 1 KSP Residual norm 3.42223<br>
> > > Linear solve converged due to CONVERGED_STEP_LENGTH iterations 1<br>
> > ><br>
> > ><br>
> > > Thank you in advance,<br>
> > > Nori<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Norihiro Watanabe<br>
><br>
><br>
><br>
><br>
> --<br>
> Norihiro Watanabe<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Norihiro Watanabe
</div>