How large are the differences? If they are at linear (or nonlinear) solver tolerance, then both are equally correct (or equally wrong).<div><br></div><div>For some codes, the domain decomposition will not be identical on restart; that changes the algorithm. Some preconditioners actually use randomness in their setup (e.g. ML). VecScatter processes messages in the order that they are received (best for performance); you can use -vecscatter_reproduce (or one of the other -vecscatter options) to process messages in deterministic order. The matrix stash (used to assemble off-process entries, usually with ADD_VALUES) does not currently have an option to reproduce exactly so finite element assembly can vary (at rounding error).</div>
<div><br></div><div>How important is bitwise reproducibility (understanding that it's expensive and limits the available algorithms) versus reproducibility to any desired tolerance?<br><br><div class="gmail_quote">On Sun, Jun 3, 2012 at 5:00 AM, Bojan Niceno <span dir="ltr"><<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Dear all,<br>
<br>
<br>
I have a rather strange issue with the incompressible Navier-Stokes
solver I am developing. It works fine with PETSc, unless I do a <i>restart</i>.
<br>
<br>
By restart I mean loading the values of dependent variables from
previously performed simulation. After such a restart, results
obtained from PETSc linear solver are slightly different than those
obtained from the previous simulation.<br>
<br>
For example, I make a first run with 100 time steps, save dependent
values at 50.<br>
<br>
I run another simulation, which starts from the results saved at 50,
but the results in the first time step in the new run will be
slightly different than those of step 51 in the previous run. The
differences are small, but still give an uneasy feeling.<br>
<br>
I suspect the difference <i>must </i>stem from different values
of: <br>
1 -dependent variables (x), or<br>
2 - sources (b), or<br>
3 - system matrices (A).<br>
<br>
In order to pinpoint the cause of the differences, I print norms of
A, x, b before and after restart. They are exactly the same which,
to my understanding, should ensure the results from a call to linear
solver are the same. Yet, that doesn't seem to be the case.<br>
<br>
Can anybody give me an advice on how to find the cause of these
differences?<br>
<br>
Is there something inside PETSc objects (such as helping vectors or
alike) which might need cleaning in order to ensure exactly the same
results after the program restart?<br>
<br>
<br>
<br>
Kind regards,<br>
<br>
<br>
Bojan<br>
</div>
</blockquote></div><br></div>