[petsc-users] sources of floating point randomness in JFNK in serial

Mark Adams mfadams at lbl.gov
Thu May 4 07:41:14 CDT 2023 

ASM is just the sub PC with one proc but gets weaker with more procs unless
you use jacobi. (maybe I am missing something).

On Thu, May 4, 2023 at 8:31 AM Mark Lohry <mlohry at gmail.com> wrote:

>  Please send the output of -snes_view.
>>
> pasted below. anything stand out?
>
>
> SNES Object: 1 MPI process
>   type: newtonls
>   maximum iterations=1, maximum function evaluations=-1
>   tolerances: relative=0.1, absolute=1e-15, solution=1e-15
>   total number of linear solver iterations=20
>   total number of function evaluations=22
>   norm schedule ALWAYS
>   Jacobian is never rebuilt
>   Jacobian is applied matrix-free with differencing
>   Preconditioning Jacobian is built using finite differences with coloring
>   SNESLineSearch Object: 1 MPI process
>     type: basic
>     maxstep=1.000000e+08, minlambda=1.000000e-12
>     tolerances: relative=1.000000e-08, absolute=1.000000e-15,
> lambda=1.000000e-08
>     maximum iterations=40
>   KSP Object: 1 MPI process
>     type: gmres
>       restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>       happy breakdown tolerance 1e-30
>     maximum iterations=20, initial guess is zero
>     tolerances:  relative=0.1, absolute=1e-15, divergence=10.
>     left preconditioning
>     using PRECONDITIONED norm type for convergence test
>   PC Object: 1 MPI process
>     type: asm
>       total subdomain blocks = 1, amount of overlap = 0
>       restriction/interpolation type - RESTRICT
>       Local solver information for first block is in the following KSP and
> PC objects on rank 0:
>       Use -ksp_view ::ascii_info_detail to display information for all
> blocks
>     KSP Object: (sub_) 1 MPI process
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (sub_) 1 MPI process
>       type: ilu
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         matrix ordering: natural
>         factor fill ratio given 1., needed 1.
>           Factored matrix follows:
>             Mat Object: (sub_) 1 MPI process
>               type: seqbaij
>               rows=16384, cols=16384, bs=16
>               package used to perform factorization: petsc
>               total: nonzeros=1277952, allocated nonzeros=1277952
>                   block size is 16
>       linear system matrix = precond matrix:
>       Mat Object: (sub_) 1 MPI process
>         type: seqbaij
>         rows=16384, cols=16384, bs=16
>         total: nonzeros=1277952, allocated nonzeros=1277952
>         total number of mallocs used during MatSetValues calls=0
>             block size is 16
>     linear system matrix followed by preconditioner matrix:
>     Mat Object: 1 MPI process
>       type: mffd
>       rows=16384, cols=16384
>         Matrix-free approximation:
>           err=1.49012e-08 (relative error in function evaluation)
>           Using wp compute h routine
>               Does not compute normU
>     Mat Object: 1 MPI process
>       type: seqbaij
>       rows=16384, cols=16384, bs=16
>       total: nonzeros=1277952, allocated nonzeros=1277952
>       total number of mallocs used during MatSetValues calls=0
>           block size is 16
>
> On Thu, May 4, 2023 at 8:30 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> If you are using MG what is the coarse grid solver?
>> -snes_view might give you that.
>>
>> On Thu, May 4, 2023 at 8:25 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Thu, May 4, 2023 at 8:21 AM Mark Lohry <mlohry at gmail.com> wrote:
>>>
>>>> Do they start very similarly and then slowly drift further apart?
>>>>
>>>>
>>>> Yes, this. I take it this sounds familiar?
>>>>
>>>> See these two examples with 20 fixed iterations pasted at the end. The
>>>> difference for one solve is slight (final SNES norm is identical to 5
>>>> digits), but in the context I'm using it in (repeated applications to solve
>>>> a steady state multigrid problem, though here just one level) the
>>>> differences add up such that I might reach global convergence in 35
>>>> iterations or 38. It's not the end of the world, but I was expecting that
>>>> with -np 1 these would be identical and I'm not sure where the root cause
>>>> would be.
>>>>
>>>
>>> The initial KSP residual is different, so its the PC. Please send the
>>> output of -snes_view. If your ASM is using direct factorization, then it
>>> could be randomness in whatever LU you are using.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>>   0 SNES Function norm 2.801842107848e+04
>>>>     0 KSP Residual norm 4.045639499595e+01
>>>>     1 KSP Residual norm 1.917999809040e+01
>>>>     2 KSP Residual norm 1.616048521958e+01
>>>> [...]
>>>>    19 KSP Residual norm 8.788043518111e-01
>>>>    20 KSP Residual norm 6.570851270214e-01
>>>>   Linear solve converged due to CONVERGED_ITS iterations 20
>>>>   1 SNES Function norm 1.801309983345e+03
>>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>>
>>>>
>>>> Same system, identical initial 0 SNES norm, 0 KSP is slightly different
>>>>
>>>>   0 SNES Function norm 2.801842107848e+04
>>>>     0 KSP Residual norm 4.045639473002e+01
>>>>     1 KSP Residual norm 1.917999883034e+01
>>>>     2 KSP Residual norm 1.616048572016e+01
>>>> [...]
>>>>    19 KSP Residual norm 8.788046348957e-01
>>>>    20 KSP Residual norm 6.570859588610e-01
>>>>   Linear solve converged due to CONVERGED_ITS iterations 20
>>>>   1 SNES Function norm 1.801311320322e+03
>>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>>
>>>> On Wed, May 3, 2023 at 11:05 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>
>>>>>
>>>>>   Do they start very similarly and then slowly drift further apart?
>>>>> That is the first couple of KSP iterations they are almost identical but
>>>>> then for each iteration get a bit further. Similar for the SNES iterations,
>>>>> starting close and then for more iterations and more solves they start
>>>>> moving apart. Or do they suddenly jump to be very different? You can run
>>>>> with -snes_monitor -ksp_monitor
>>>>>
>>>>> On May 3, 2023, at 9:07 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>>
>>>>> This is on a single MPI rank. I haven't checked the coloring, was just
>>>>> guessing there. But the solutions/residuals are slightly different from run
>>>>> to run.
>>>>>
>>>>> Fair to say that for serial JFNK/asm ilu0/gmres we should expect
>>>>> bitwise identical results?
>>>>>
>>>>>
>>>>> On Wed, May 3, 2023, 8:50 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>>
>>>>>>
>>>>>>   No, the coloring should be identical every time. Do you see
>>>>>> differences with 1 MPI rank? (Or much smaller ones?).
>>>>>>
>>>>>>
>>>>>>
>>>>>> > On May 3, 2023, at 8:42 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>>> >
>>>>>> > I'm running multiple iterations of newtonls with an MFFD/JFNK
>>>>>> nonlinear solver where I give it the sparsity. PC asm, KSP gmres, with
>>>>>> SNESSetLagJacobian -2 (compute once and then frozen jacobian).
>>>>>> >
>>>>>> > I'm seeing slight (<1%) but nonzero differences in residuals from
>>>>>> run to run. I'm wondering where randomness might enter here -- does the
>>>>>> jacobian coloring use a random seed?
>>>>>>
>>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230504/09fc8173/attachment.html>

More information about the petsc-users mailing list