[petsc-users] sources of floating point randomness in JFNK in serial
Matthew Knepley
knepley at gmail.com
Thu May 4 07:34:59 CDT 2023
On Thu, May 4, 2023 at 8:31 AM Mark Lohry <mlohry at gmail.com> wrote:
> Please send the output of -snes_view.
>>
> pasted below. anything stand out?
>
Try -pc_type none. If the first KSP residual is different, then it is
something in the MatMFFD, and then I would
guess it is something variable in the residual evaluation routine. That
should be easy to check by printing norms
of the residual each time it is evaluated.
Thanks,
Matt
> SNES Object: 1 MPI process
> type: newtonls
> maximum iterations=1, maximum function evaluations=-1
> tolerances: relative=0.1, absolute=1e-15, solution=1e-15
> total number of linear solver iterations=20
> total number of function evaluations=22
> norm schedule ALWAYS
> Jacobian is never rebuilt
> Jacobian is applied matrix-free with differencing
> Preconditioning Jacobian is built using finite differences with coloring
> SNESLineSearch Object: 1 MPI process
> type: basic
> maxstep=1.000000e+08, minlambda=1.000000e-12
> tolerances: relative=1.000000e-08, absolute=1.000000e-15,
> lambda=1.000000e-08
> maximum iterations=40
> KSP Object: 1 MPI process
> type: gmres
> restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> happy breakdown tolerance 1e-30
> maximum iterations=20, initial guess is zero
> tolerances: relative=0.1, absolute=1e-15, divergence=10.
> left preconditioning
> using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI process
> type: asm
> total subdomain blocks = 1, amount of overlap = 0
> restriction/interpolation type - RESTRICT
> Local solver information for first block is in the following KSP and
> PC objects on rank 0:
> Use -ksp_view ::ascii_info_detail to display information for all
> blocks
> KSP Object: (sub_) 1 MPI process
> type: preonly
> maximum iterations=10000, initial guess is zero
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
> left preconditioning
> using NONE norm type for convergence test
> PC Object: (sub_) 1 MPI process
> type: ilu
> out-of-place factorization
> 0 levels of fill
> tolerance for zero pivot 2.22045e-14
> matrix ordering: natural
> factor fill ratio given 1., needed 1.
> Factored matrix follows:
> Mat Object: (sub_) 1 MPI process
> type: seqbaij
> rows=16384, cols=16384, bs=16
> package used to perform factorization: petsc
> total: nonzeros=1277952, allocated nonzeros=1277952
> block size is 16
> linear system matrix = precond matrix:
> Mat Object: (sub_) 1 MPI process
> type: seqbaij
> rows=16384, cols=16384, bs=16
> total: nonzeros=1277952, allocated nonzeros=1277952
> total number of mallocs used during MatSetValues calls=0
> block size is 16
> linear system matrix followed by preconditioner matrix:
> Mat Object: 1 MPI process
> type: mffd
> rows=16384, cols=16384
> Matrix-free approximation:
> err=1.49012e-08 (relative error in function evaluation)
> Using wp compute h routine
> Does not compute normU
> Mat Object: 1 MPI process
> type: seqbaij
> rows=16384, cols=16384, bs=16
> total: nonzeros=1277952, allocated nonzeros=1277952
> total number of mallocs used during MatSetValues calls=0
> block size is 16
>
> On Thu, May 4, 2023 at 8:30 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> If you are using MG what is the coarse grid solver?
>> -snes_view might give you that.
>>
>> On Thu, May 4, 2023 at 8:25 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Thu, May 4, 2023 at 8:21 AM Mark Lohry <mlohry at gmail.com> wrote:
>>>
>>>> Do they start very similarly and then slowly drift further apart?
>>>>
>>>>
>>>> Yes, this. I take it this sounds familiar?
>>>>
>>>> See these two examples with 20 fixed iterations pasted at the end. The
>>>> difference for one solve is slight (final SNES norm is identical to 5
>>>> digits), but in the context I'm using it in (repeated applications to solve
>>>> a steady state multigrid problem, though here just one level) the
>>>> differences add up such that I might reach global convergence in 35
>>>> iterations or 38. It's not the end of the world, but I was expecting that
>>>> with -np 1 these would be identical and I'm not sure where the root cause
>>>> would be.
>>>>
>>>
>>> The initial KSP residual is different, so its the PC. Please send the
>>> output of -snes_view. If your ASM is using direct factorization, then it
>>> could be randomness in whatever LU you are using.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> 0 SNES Function norm 2.801842107848e+04
>>>> 0 KSP Residual norm 4.045639499595e+01
>>>> 1 KSP Residual norm 1.917999809040e+01
>>>> 2 KSP Residual norm 1.616048521958e+01
>>>> [...]
>>>> 19 KSP Residual norm 8.788043518111e-01
>>>> 20 KSP Residual norm 6.570851270214e-01
>>>> Linear solve converged due to CONVERGED_ITS iterations 20
>>>> 1 SNES Function norm 1.801309983345e+03
>>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>>
>>>>
>>>> Same system, identical initial 0 SNES norm, 0 KSP is slightly different
>>>>
>>>> 0 SNES Function norm 2.801842107848e+04
>>>> 0 KSP Residual norm 4.045639473002e+01
>>>> 1 KSP Residual norm 1.917999883034e+01
>>>> 2 KSP Residual norm 1.616048572016e+01
>>>> [...]
>>>> 19 KSP Residual norm 8.788046348957e-01
>>>> 20 KSP Residual norm 6.570859588610e-01
>>>> Linear solve converged due to CONVERGED_ITS iterations 20
>>>> 1 SNES Function norm 1.801311320322e+03
>>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>>
>>>> On Wed, May 3, 2023 at 11:05 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>
>>>>>
>>>>> Do they start very similarly and then slowly drift further apart?
>>>>> That is the first couple of KSP iterations they are almost identical but
>>>>> then for each iteration get a bit further. Similar for the SNES iterations,
>>>>> starting close and then for more iterations and more solves they start
>>>>> moving apart. Or do they suddenly jump to be very different? You can run
>>>>> with -snes_monitor -ksp_monitor
>>>>>
>>>>> On May 3, 2023, at 9:07 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>>
>>>>> This is on a single MPI rank. I haven't checked the coloring, was just
>>>>> guessing there. But the solutions/residuals are slightly different from run
>>>>> to run.
>>>>>
>>>>> Fair to say that for serial JFNK/asm ilu0/gmres we should expect
>>>>> bitwise identical results?
>>>>>
>>>>>
>>>>> On Wed, May 3, 2023, 8:50 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>>
>>>>>>
>>>>>> No, the coloring should be identical every time. Do you see
>>>>>> differences with 1 MPI rank? (Or much smaller ones?).
>>>>>>
>>>>>>
>>>>>>
>>>>>> > On May 3, 2023, at 8:42 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>>> >
>>>>>> > I'm running multiple iterations of newtonls with an MFFD/JFNK
>>>>>> nonlinear solver where I give it the sparsity. PC asm, KSP gmres, with
>>>>>> SNESSetLagJacobian -2 (compute once and then frozen jacobian).
>>>>>> >
>>>>>> > I'm seeing slight (<1%) but nonzero differences in residuals from
>>>>>> run to run. I'm wondering where randomness might enter here -- does the
>>>>>> jacobian coloring use a random seed?
>>>>>>
>>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230504/efcabbd0/attachment-0001.html>
More information about the petsc-users
mailing list