<div dir="ltr">OK, will do, thanks.<div><br></div><div>David</div><div><br></div><div><br></div><div><br></div><div>On Wed, Jan 13, 2016 at 4:05 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Since you are using a direct solver almost for sure a bug in your parallel function or parallel Jacobian.<br>
<br>
Try -snes_mf_operator try -snes_fd try -snes_type test as three different approaches to see what is going on.<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
> On Jan 13, 2016, at 2:51 PM, David Knezevic <<a href="mailto:david.knezevic@akselos.com">david.knezevic@akselos.com</a>> wrote:<br>
><br>
> Oops! I pasted the wrong text for the serial case. The correct text is below:<br>
><br>
> Serial case:<br>
> NL step 0, |residual|_2 = 4.714515e-02<br>
> Line search: gnorm after quadratic fit 7.862867755130e-02<br>
> Line search: Cubically determined step, current gnorm 4.663945044088e-02 lambda=1.4276549223307832e-02<br>
> NL step 1, |residual|_2 = 4.663945e-02<br>
> Line search: gnorm after quadratic fit 6.977268532963e-02<br>
> Line search: Cubically determined step, current gnorm 4.594912791877e-02 lambda=2.3644826349821228e-02<br>
> NL step 2, |residual|_2 = 4.594913e-02<br>
> Line search: gnorm after quadratic fit 5.502067915588e-02<br>
> Line search: Cubically determined step, current gnorm 4.494531287593e-02 lambda=4.1260496881982515e-02<br>
> NL step 3, |residual|_2 = 4.494531e-02<br>
> Line search: gnorm after quadratic fit 5.415371014813e-02<br>
> Line search: Cubically determined step, current gnorm 4.392165909219e-02 lambda=3.6375617606865668e-02<br>
> NL step 4, |residual|_2 = 4.392166e-02<br>
> Line search: gnorm after quadratic fit 4.631663907262e-02<br>
> Line search: Cubically determined step, current gnorm 4.246200768767e-02 lambda=5.0000000000000003e-02<br>
> NL step 5, |residual|_2 = 4.246201e-02<br>
> Line search: gnorm after quadratic fit 4.222105256158e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 6, |residual|_2 = 4.222105e-02<br>
> Line search: gnorm after quadratic fit 4.026081168915e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 7, |residual|_2 = 4.026081e-02<br>
> Line search: gnorm after quadratic fit 3.776439443011e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 8, |residual|_2 = 3.776439e-02<br>
> Line search: gnorm after quadratic fit 3.659796213553e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 9, |residual|_2 = 3.659796e-02<br>
> Line search: gnorm after quadratic fit 3.423207563496e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 10, |residual|_2 = 3.423208e-02<br>
> Line search: gnorm after quadratic fit 3.116928356075e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 11, |residual|_2 = 3.116928e-02<br>
> Line search: gnorm after quadratic fit 2.874310673331e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 12, |residual|_2 = 2.874311e-02<br>
> Line search: gnorm after quadratic fit 2.587826447631e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 13, |residual|_2 = 2.587826e-02<br>
> Line search: gnorm after quadratic fit 2.344160918669e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 14, |residual|_2 = 2.344161e-02<br>
> Line search: gnorm after quadratic fit 2.187719801063e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 15, |residual|_2 = 2.187720e-02<br>
> Line search: gnorm after quadratic fit 1.983089025936e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 16, |residual|_2 = 1.983089e-02<br>
> Line search: gnorm after quadratic fit 1.791227696650e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 17, |residual|_2 = 1.791228e-02<br>
> Line search: gnorm after quadratic fit 1.613250592206e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 18, |residual|_2 = 1.613251e-02<br>
> Line search: gnorm after quadratic fit 1.455841890804e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 19, |residual|_2 = 1.455842e-02<br>
> Line search: gnorm after quadratic fit 1.321849665170e-02<br>
> Line search: Quadratically determined step, lambda=1.0574900347563776e-01<br>
> NL step 20, |residual|_2 = 1.321850e-02<br>
> Line search: gnorm after quadratic fit 9.209642717528e-03<br>
> Line search: Quadratically determined step, lambda=3.0589679103560180e-01<br>
> NL step 21, |residual|_2 = 9.209643e-03<br>
> Line search: gnorm after quadratic fit 7.590944125425e-03<br>
> Line search: Quadratically determined step, lambda=2.0920307644146574e-01<br>
> NL step 22, |residual|_2 = 7.590944e-03<br>
> Line search: gnorm after quadratic fit 4.373921456388e-03<br>
> Line search: Quadratically determined step, lambda=4.2379743756255861e-01<br>
> NL step 23, |residual|_2 = 4.373921e-03<br>
> Line search: gnorm after quadratic fit 3.681355014898e-03<br>
> Line search: Quadratically determined step, lambda=1.9626628361883081e-01<br>
> NL step 24, |residual|_2 = 3.681355e-03<br>
> Line search: gnorm after quadratic fit 2.594785108727e-03<br>
> Line search: Quadratically determined step, lambda=3.8057573229158653e-01<br>
> NL step 25, |residual|_2 = 2.594785e-03<br>
> Line search: gnorm after quadratic fit 1.803191839408e-03<br>
> Line search: Quadratically determined step, lambda=4.3574150080610474e-01<br>
> NL step 26, |residual|_2 = 1.803192e-03<br>
> Line search: Using full step: fnorm 1.803191839408e-03 gnorm 9.015954497317e-04<br>
> NL step 27, |residual|_2 = 9.015954e-04<br>
> Line search: Using full step: fnorm 9.015954497317e-04 gnorm 1.390181456520e-13<br>
> NL step 28, |residual|_2 = 1.390181e-13<br>
> Number of nonlinear iterations: 28<br>
><br>
><br>
> On Wed, Jan 13, 2016 at 3:48 PM, David Knezevic <<a href="mailto:david.knezevic@akselos.com">david.knezevic@akselos.com</a>> wrote:<br>
> I'm using NEWTONLS (with mumps for the linear solves) to do a nonlinear PDE solve. It converges well when I use 1 core. When I use 2 or more cores, the line search stagnates. I've pasted the output of -snes_linesearch_monitor below in these two cases.<br>
><br>
> I was wondering if this implies that I must have a bug in parallel, or if perhaps the NEWTONLS solver can behave slightly differently in parallel?<br>
><br>
> Thanks,<br>
> David<br>
><br>
> ---------------------------------------------------------------------------------------<br>
><br>
><br>
><br>
> Parallel case:<br>
> NL step 0, |residual|_2 = 4.714515e-02<br>
> Line search: gnorm after quadratic fit 7.862867755323e-02<br>
> Line search: Cubically determined step, current gnorm 4.663945043239e-02 lambda=1.4276549921126183e-02<br>
> NL step 1, |residual|_2 = 4.663945e-02<br>
> Line search: gnorm after quadratic fit 6.977268575068e-02<br>
> Line search: Cubically determined step, current gnorm 4.594912794004e-02 lambda=2.3644825912085998e-02<br>
> NL step 2, |residual|_2 = 4.594913e-02<br>
> Line search: gnorm after quadratic fit 5.502067932478e-02<br>
> Line search: Cubically determined step, current gnorm 4.494531294405e-02 lambda=4.1260497615261321e-02<br>
> NL step 3, |residual|_2 = 4.494531e-02<br>
> Line search: gnorm after quadratic fit 5.415371063247e-02<br>
> Line search: Cubically determined step, current gnorm 4.392165925471e-02 lambda=3.6375618871780056e-02<br>
> NL step 4, |residual|_2 = 4.392166e-02<br>
> Line search: gnorm after quadratic fit 4.631663976615e-02<br>
> Line search: Cubically determined step, current gnorm 4.246200798775e-02 lambda=5.0000000000000003e-02<br>
> NL step 5, |residual|_2 = 4.246201e-02<br>
> Line search: gnorm after quadratic fit 4.222105321728e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 6, |residual|_2 = 4.222105e-02<br>
> Line search: gnorm after quadratic fit 4.026081251872e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 7, |residual|_2 = 4.026081e-02<br>
> Line search: gnorm after quadratic fit 3.776439532346e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 8, |residual|_2 = 3.776440e-02<br>
> Line search: gnorm after quadratic fit 3.659796311121e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 9, |residual|_2 = 3.659796e-02<br>
> Line search: gnorm after quadratic fit 3.423207664901e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 10, |residual|_2 = 3.423208e-02<br>
> Line search: gnorm after quadratic fit 3.116928452225e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 11, |residual|_2 = 3.116928e-02<br>
> Line search: gnorm after quadratic fit 2.874310955274e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 12, |residual|_2 = 2.874311e-02<br>
> Line search: gnorm after quadratic fit 2.587826662305e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 13, |residual|_2 = 2.587827e-02<br>
> Line search: gnorm after quadratic fit 2.344161073075e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 14, |residual|_2 = 2.344161e-02<br>
> Line search: gnorm after quadratic fit 2.187719889554e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 15, |residual|_2 = 2.187720e-02<br>
> Line search: gnorm after quadratic fit 1.983089075086e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 16, |residual|_2 = 1.983089e-02<br>
> Line search: gnorm after quadratic fit 1.791227711151e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 17, |residual|_2 = 1.791228e-02<br>
> Line search: gnorm after quadratic fit 1.613250573900e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 18, |residual|_2 = 1.613251e-02<br>
> Line search: gnorm after quadratic fit 1.455841843183e-02<br>
> Line search: Quadratically determined step, lambda=1.0000000000000001e-01<br>
> NL step 19, |residual|_2 = 1.455842e-02<br>
> Line search: gnorm after quadratic fit 1.321849780208e-02<br>
> Line search: Quadratically determined step, lambda=1.0574876450981290e-01<br>
> NL step 20, |residual|_2 = 1.321850e-02<br>
> Line search: gnorm after quadratic fit 9.209641609489e-03<br>
> Line search: Quadratically determined step, lambda=3.0589684959139674e-01<br>
> NL step 21, |residual|_2 = 9.209642e-03<br>
> Line search: gnorm after quadratic fit 7.590942028574e-03<br>
> Line search: Quadratically determined step, lambda=2.0920305898507460e-01<br>
> NL step 22, |residual|_2 = 7.590942e-03<br>
> Line search: gnorm after quadratic fit 4.373918927227e-03<br>
> Line search: Quadratically determined step, lambda=4.2379743128074154e-01<br>
> NL step 23, |residual|_2 = 4.373919e-03<br>
> Line search: gnorm after quadratic fit 3.681351665911e-03<br>
> Line search: Quadratically determined step, lambda=1.9626618428089049e-01<br>
> NL step 24, |residual|_2 = 3.681352e-03<br>
> Line search: gnorm after quadratic fit 2.594782418891e-03<br>
> Line search: Quadratically determined step, lambda=3.8057533372167579e-01<br>
> NL step 25, |residual|_2 = 2.594782e-03<br>
> Line search: gnorm after quadratic fit 1.803188279452e-03<br>
> Line search: Quadratically determined step, lambda=4.3574109448916826e-01<br>
> NL step 26, |residual|_2 = 1.803188e-03<br>
> Line search: Using full step: fnorm 1.803188279452e-03 gnorm 9.015947319176e-04<br>
> NL step 27, |residual|_2 = 9.015947e-04<br>
> Line search: Using full step: fnorm 9.015947319176e-04 gnorm 7.088879385731e-08<br>
> NL step 28, |residual|_2 = 7.088879e-08<br>
> Line search: gnorm after quadratic fit 7.088878906502e-08<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088878957116e-08 lambda=2.1132490715284968e-01<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385683e-08 lambda=9.2196195824189087e-02<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385711e-08 lambda=4.0004532931495446e-02<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385722e-08 lambda=1.7374764617622523e-02<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385726e-08 lambda=7.5449542135114234e-03<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=3.2764749100364717e-03<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.4228361655588414e-03<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=6.1787884492365153e-04<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=2.6831916265377548e-04<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.1651988987471248e-04<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=5.0599757911789984e-05<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=2.1973377296845284e-05<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=9.5421268734746417e-06<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=4.1437501409853001e-06<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.7994589108402447e-06<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=7.8143041004756041e-07<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=3.3934283359762142e-07<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.4736252548330828e-07<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=6.3993436038104693e-08<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=2.7789696481734489e-08<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.2067913185456743e-08<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=5.2405944320925838e-09<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=2.2757729177880525e-09<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=9.8827383810151057e-10<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=4.2916635989551390e-10<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.8636915940199893e-10<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=8.0932400164504977e-11<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=3.5145586412497970e-11<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.5262271250668997e-11<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=6.6277717206096633e-12<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=2.8781665100197773e-12<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=1.2498684035299616e-12<br>
> Line search: Cubic step no good, shrinking lambda, current gnorm 7.088879385731e-08 lambda=5.4276603549660526e-13<br>
> Line search: unable to find good step length! After 33 tries<br>
> Line search: fnorm=7.0888793857309783e-08, gnorm=7.0888793857309783e-08, ynorm=2.4650076775058285e-08, minlambda=9.9999999999999998e-13, lambda=5.4276603549660526e-13, initial slope=-5.0252210945441613e-15<br>
><br>
><br>
<br>
</div></div></blockquote></div><br></div></div>