<div dir="ltr"><div>Hi Barry,<br><br>Is it possible that -snes_mf_operator makes convergence of linear
solves slower if unknowns are poorly scaled for multiphysics problems? I gave up to check Jacobian for the large problem because it takes too
long time. Instead I tested it with several different small size
problems and noticed scaling of my unknowns makes a difference in FD Jacobian. My unknowns are two kinds: pressure (1e7) and temperature (1e3). I scaled pressure by 1e-5 and got the following different results from -snes_type test<br>
<br>without scaling <br>Norm of matrix ratio 3.81371e-05 difference 6.42349e+06 (user-defined state)<br><br>with scaling<br>Norm of matrix ratio 8.69182e-09 difference 1463.98 (user-defined state)<br><br></div>which may suggest that a differentiate parameter h is not properly set if unknowns are poorly scaled. I also tested the scaling for the large problem with keeping fluid properties constant and got the following different convergence behaviours:<br>
<div><div><br>without -snes_mf_operator<br> 0 SNES Function norm 4.425457683773e+04 <br> 0 KSP Residual norm 44254.6 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
1 KSP Residual norm 0.000168321 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 2<br> 2 KSP Residual norm 8.18977e-06 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br>
Linear solve converged due to CONVERGED_RTOL iterations 2<br> 3 KSP Residual norm 4.75882e-07 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
4 KSP Residual norm 4.06286e-08 <br> Linear solve converged due to CONVERGED_RTOL iterations 4<br> 1 SNES Function norm 2.229156139237e+05 <br><br>with -snes_mf_operator<br> 0 SNES Function norm 4.425457683883e+04 <br>
0 KSP Residual norm 44254.6 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 2<br> 1 KSP Residual norm 5255.66 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br>
Linear solve converged due to CONVERGED_RTOL iterations 1<br> 2 KSP Residual norm 1646.58 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
3 KSP Residual norm 1518.05 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br>...<br> 42 KSP Residual norm 0.656962 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br>
Linear solve converged due to CONVERGED_RTOL iterations 1<br> 43 KSP Residual norm 0.462202 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
<br>with -snes_mf_operator and scaling of pressure by 1e-5<br> 0 SNES Function norm 4.425457683773e+04 <br> 0 KSP Residual norm 44254.6 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 2<br>
1 KSP Residual norm 1883.94 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br> 2 KSP Residual norm 893.88 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br>
Linear solve converged due to CONVERGED_RTOL iterations 1<br>...<br> 42 KSP Residual norm 6.66081e-08 <br> Linear solve converged due to CONVERGED_ITS iterations 1<br> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
43 KSP Residual norm 2.17062e-08 <br> Linear solve converged due to CONVERGED_RTOL iterations 43<br> 1 SNES Function norm 2.200867439822e+05 <br><br><br><br><br><br><br></div></div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Mon, Apr 28, 2014 at 5:59 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
It will take a very long time<br>
<div class="HOEnZb"><div class="h5"><br>
On Apr 28, 2014, at 9:14 AM, Norihiro Watanabe <<a href="mailto:norihiro.w@gmail.com">norihiro.w@gmail.com</a>> wrote:<br>
<br>
> I cannot surely say my Jacobian for this particular problem is correct, as I have not checked it. For a smaller problem, I've already checked its correctness using -snes_type test or -snes_compare_explicit (but linear solve and nonlinear solve with FD one need a few more iterations than with my Jacobian). To make it sure, now I started -snes_type test for the problem and will update you once it finished. By the way, I'm waiting the calculation for more than three hours now. Is it usual for a large problem (>1e6 dof) or is there something wrong?<br>
><br>
><br>
><br>
><br>
><br>
> On Mon, Apr 28, 2014 at 6:34 AM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> I have run your code. I changed to use -snes_type newtonls and also -snes_mf_operator there is something wrong with your Jacobian:<br>
><br>
> Without -snes_mf_operator<br>
> 0 SNES Function norm 1.821611413735e+03<br>
> 0 KSP Residual norm 1821.61<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 1 KSP Residual norm 0.000167024<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 2 KSP Residual norm 7.66595e-06<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 3 KSP Residual norm 4.4581e-07<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 4 KSP Residual norm 3.77537e-08<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 5 KSP Residual norm 2.20453e-09<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 6 KSP Residual norm 1.711e-10<br>
> Linear solve converged due to CONVERGED_RTOL iterations 6<br>
><br>
> with -snes_mf_operator<br>
><br>
> 0 SNES Function norm 1.821611413735e+03<br>
> 0 KSP Residual norm 1821.61<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 1 KSP Residual norm 1796.39<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 2 KSP Residual norm 1786.2<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 3 KSP Residual norm 1741.11<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 4 KSP Residual norm 1733.92<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 5 KSP Residual norm 1726.57<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 6 KSP Residual norm 1725.35<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 7 KSP Residual norm 1723.89<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 8 KSP Residual norm 1715.41<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 9 KSP Residual norm 1713.72<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
> Linear solve converged due to CONVERGED_RTOL iterations 1<br>
> 10 KSP Residual norm 1702.84<br>
> Linear solve converged due to CONVERGED_ITS iterations 1<br>
><br>
> …<br>
><br>
> This means your Jacobian is wrong. Your first order of business is to fix your Jacobian. I noticed in previous emails your discussion with Jed about switching to MatGetLocalSubMatrix() and using -snes_type test YOU NEED TO DO THIS. You will get no where with an incorrect Jacobian. You need to fix your Jacobian before you do anything else! No amount of other options or methods will help you with a wrong Jacobian! Once you have a correct Jacobian if you still have convergence problems let us know and we can make further suggestions.<br>
><br>
> Barry<br>
><br>
> On Apr 25, 2014, at 7:31 AM, Norihiro Watanabe <<a href="mailto:norihiro.w@gmail.com">norihiro.w@gmail.com</a>> wrote:<br>
><br>
> > Hi,<br>
> ><br>
> > In my simulation, nonlinear solve with the trust regtion method got stagnent after linear solve (see output below). Is it possible that the method goes to inifite loop? Is there any parameter to avoid this situation?<br>
> ><br>
> > 0 SNES Function norm 1.828728087153e+03<br>
> > 0 KSP Residual norm 91.2735<br>
> > Linear solve converged due to CONVERGED_ITS iterations 1<br>
> > Linear solve converged due to CONVERGED_RTOL iterations 3<br>
> > 1 KSP Residual norm 3.42223<br>
> > Linear solve converged due to CONVERGED_STEP_LENGTH iterations 1<br>
> ><br>
> ><br>
> > Thank you in advance,<br>
> > Nori<br>
><br>
><br>
><br>
><br>
> --<br>
> Norihiro Watanabe<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Norihiro Watanabe
</div>