[petsc-users] Solutions different after program restart

[Niceno Bojan]

Dear Jed,


thanks for the answer.

> How large are the differences? If they are at linear (or nonlinear) solver
> tolerance, then both are equally correct (or equally wrong).

Good point!  For example:

Residual norm obtained from a call to "KSPGetResidualNorm(ksp, &rnorm);", gives:

- First run: 1.98077e-02
- Second run: 2.61329e-02

The tolerances I request from KSP's gmres solver are:

- Linear solver relative tolerance = 0.001;
- Linear solver absolute tolerance = 1.0e-5;

Does it tell you something?  It does not to me :-(


> For some codes, the domain decomposition will not be identical on restart;
> that changes the algorithm. 

That is clear, but what I am comparing are two sequential runs.


    Kind regards,

  
    Bojan


Some preconditioners actually use randomness in
their setup (e.g. ML). VecScatter processes messages in the order that they
are received (best for performance); you can use -vecscatter_reproduce (or
one of the other -vecscatter options) to process messages in deterministic
order. The matrix stash (used to assemble off-process entries, usually with
ADD_VALUES) does not currently have an option to reproduce exactly so
finite element assembly can vary (at rounding error).

How important is bitwise reproducibility (understanding that it's expensive
and limits the available algorithms) versus reproducibility to any desired
tolerance?

On Sun, Jun 3, 2012 at 5:00 AM, Bojan Niceno <bojan.niceno at psi.ch> wrote:

>  Dear all,
>
>
> I have a rather strange issue with the incompressible Navier-Stokes solver
> I am developing.  It works fine with PETSc, unless I do a *restart*.
>
> By restart I mean loading the values of dependent variables from
> previously performed simulation.  After such a restart, results obtained
> from PETSc linear solver are slightly different than those obtained from
> the previous simulation.
>
> For example, I make a first run with 100 time steps, save dependent values
> at 50.
>
> I run another simulation, which starts from the results saved at 50, but
> the results in the first time step in the new run will be slightly
> different than those of step 51 in the previous run.  The differences are
> small, but still give an uneasy feeling.
>
> I suspect the difference *must *stem from different values of:
> 1 -dependent variables (x), or
> 2 - sources (b), or
> 3 - system matrices (A).
>
> In order to pinpoint the cause of the differences, I print norms of  A, x,
> b before and after restart.  They are exactly the same which, to my
> understanding, should ensure the results from a call to linear solver are
> the same.  Yet, that doesn't seem to be the case.
>
> Can anybody give me an advice on how to find the cause of these
> differences?
>
> Is there something inside PETSc objects (such as helping vectors or alike)
> which might need cleaning in order to ensure exactly the same results after
> the program restart?
>
>
>
>     Kind regards,
>
>
>     Bojan
>