[petsc-users] sources of floating point randomness in JFNK in serial
Mark Lohry
mlohry at gmail.com
Thu May 4 07:31:45 CDT 2023
>
> Please send the output of -snes_view.
>
pasted below. anything stand out?
SNES Object: 1 MPI process
type: newtonls
maximum iterations=1, maximum function evaluations=-1
tolerances: relative=0.1, absolute=1e-15, solution=1e-15
total number of linear solver iterations=20
total number of function evaluations=22
norm schedule ALWAYS
Jacobian is never rebuilt
Jacobian is applied matrix-free with differencing
Preconditioning Jacobian is built using finite differences with coloring
SNESLineSearch Object: 1 MPI process
type: basic
maxstep=1.000000e+08, minlambda=1.000000e-12
tolerances: relative=1.000000e-08, absolute=1.000000e-15,
lambda=1.000000e-08
maximum iterations=40
KSP Object: 1 MPI process
type: gmres
restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
happy breakdown tolerance 1e-30
maximum iterations=20, initial guess is zero
tolerances: relative=0.1, absolute=1e-15, divergence=10.
left preconditioning
using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI process
type: asm
total subdomain blocks = 1, amount of overlap = 0
restriction/interpolation type - RESTRICT
Local solver information for first block is in the following KSP and
PC objects on rank 0:
Use -ksp_view ::ascii_info_detail to display information for all
blocks
KSP Object: (sub_) 1 MPI process
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (sub_) 1 MPI process
type: ilu
out-of-place factorization
0 levels of fill
tolerance for zero pivot 2.22045e-14
matrix ordering: natural
factor fill ratio given 1., needed 1.
Factored matrix follows:
Mat Object: (sub_) 1 MPI process
type: seqbaij
rows=16384, cols=16384, bs=16
package used to perform factorization: petsc
total: nonzeros=1277952, allocated nonzeros=1277952
block size is 16
linear system matrix = precond matrix:
Mat Object: (sub_) 1 MPI process
type: seqbaij
rows=16384, cols=16384, bs=16
total: nonzeros=1277952, allocated nonzeros=1277952
total number of mallocs used during MatSetValues calls=0
block size is 16
linear system matrix followed by preconditioner matrix:
Mat Object: 1 MPI process
type: mffd
rows=16384, cols=16384
Matrix-free approximation:
err=1.49012e-08 (relative error in function evaluation)
Using wp compute h routine
Does not compute normU
Mat Object: 1 MPI process
type: seqbaij
rows=16384, cols=16384, bs=16
total: nonzeros=1277952, allocated nonzeros=1277952
total number of mallocs used during MatSetValues calls=0
block size is 16
On Thu, May 4, 2023 at 8:30 AM Mark Adams <mfadams at lbl.gov> wrote:
> If you are using MG what is the coarse grid solver?
> -snes_view might give you that.
>
> On Thu, May 4, 2023 at 8:25 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, May 4, 2023 at 8:21 AM Mark Lohry <mlohry at gmail.com> wrote:
>>
>>> Do they start very similarly and then slowly drift further apart?
>>>
>>>
>>> Yes, this. I take it this sounds familiar?
>>>
>>> See these two examples with 20 fixed iterations pasted at the end. The
>>> difference for one solve is slight (final SNES norm is identical to 5
>>> digits), but in the context I'm using it in (repeated applications to solve
>>> a steady state multigrid problem, though here just one level) the
>>> differences add up such that I might reach global convergence in 35
>>> iterations or 38. It's not the end of the world, but I was expecting that
>>> with -np 1 these would be identical and I'm not sure where the root cause
>>> would be.
>>>
>>
>> The initial KSP residual is different, so its the PC. Please send the
>> output of -snes_view. If your ASM is using direct factorization, then it
>> could be randomness in whatever LU you are using.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> 0 SNES Function norm 2.801842107848e+04
>>> 0 KSP Residual norm 4.045639499595e+01
>>> 1 KSP Residual norm 1.917999809040e+01
>>> 2 KSP Residual norm 1.616048521958e+01
>>> [...]
>>> 19 KSP Residual norm 8.788043518111e-01
>>> 20 KSP Residual norm 6.570851270214e-01
>>> Linear solve converged due to CONVERGED_ITS iterations 20
>>> 1 SNES Function norm 1.801309983345e+03
>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>
>>>
>>> Same system, identical initial 0 SNES norm, 0 KSP is slightly different
>>>
>>> 0 SNES Function norm 2.801842107848e+04
>>> 0 KSP Residual norm 4.045639473002e+01
>>> 1 KSP Residual norm 1.917999883034e+01
>>> 2 KSP Residual norm 1.616048572016e+01
>>> [...]
>>> 19 KSP Residual norm 8.788046348957e-01
>>> 20 KSP Residual norm 6.570859588610e-01
>>> Linear solve converged due to CONVERGED_ITS iterations 20
>>> 1 SNES Function norm 1.801311320322e+03
>>> Nonlinear solve converged due to CONVERGED_ITS iterations 1
>>>
>>> On Wed, May 3, 2023 at 11:05 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>
>>>>
>>>> Do they start very similarly and then slowly drift further apart?
>>>> That is the first couple of KSP iterations they are almost identical but
>>>> then for each iteration get a bit further. Similar for the SNES iterations,
>>>> starting close and then for more iterations and more solves they start
>>>> moving apart. Or do they suddenly jump to be very different? You can run
>>>> with -snes_monitor -ksp_monitor
>>>>
>>>> On May 3, 2023, at 9:07 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>
>>>> This is on a single MPI rank. I haven't checked the coloring, was just
>>>> guessing there. But the solutions/residuals are slightly different from run
>>>> to run.
>>>>
>>>> Fair to say that for serial JFNK/asm ilu0/gmres we should expect
>>>> bitwise identical results?
>>>>
>>>>
>>>> On Wed, May 3, 2023, 8:50 PM Barry Smith <bsmith at petsc.dev> wrote:
>>>>
>>>>>
>>>>> No, the coloring should be identical every time. Do you see
>>>>> differences with 1 MPI rank? (Or much smaller ones?).
>>>>>
>>>>>
>>>>>
>>>>> > On May 3, 2023, at 8:42 PM, Mark Lohry <mlohry at gmail.com> wrote:
>>>>> >
>>>>> > I'm running multiple iterations of newtonls with an MFFD/JFNK
>>>>> nonlinear solver where I give it the sparsity. PC asm, KSP gmres, with
>>>>> SNESSetLagJacobian -2 (compute once and then frozen jacobian).
>>>>> >
>>>>> > I'm seeing slight (<1%) but nonzero differences in residuals from
>>>>> run to run. I'm wondering where randomness might enter here -- does the
>>>>> jacobian coloring use a random seed?
>>>>>
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230504/a88f25ba/attachment.html>
More information about the petsc-users
mailing list