Non repeatability issue

[Aldo Bonfiglioli]

Dear all,
this is a follow-up to an old mail concerning non-repeatibility
issues in a parallel environment.

We are solving the steady 3D RANS eqns using
Newtons's algorithm. All equations (turbulence included)
are fully coupled.

Our non-linear convergence history shows remarkable
non-repeatibility among subsequent parallel runs
(see the enclosed plot referred to as MeTis partitioning).

We believe (but of course cannot rule out other
reasons) that this is due to the fact that
within those "ghosted" nodes that are shared by more
 than two sub-domains the rhs will be slightly
different among subsequent runs
depending on the order by which contributions
are accumulated in these interfacial nodes.
Since we push convergence down to machine accuracy,
this may not be irrelevant.

We then devised an alternative partitioning
(though applicable only on very simple geometries)
that guarantees that "ghosted" nodes are shared
by not more than two procs. This guarantees that
not more than two contributions will be accumulated
in the ghosted nodes.
Using this partitioning (referred to as geometric
in the enclosed plots)
subsequent runs give indeed identical results.

In all cases, of course, we start from identical initial
solutions.

A colleague of ours has suggested the following:

> This could be a problem of GMRES least-squares
> system being poorly conditioned. I would try set a higher (very high in
> fact) the tolerance for the inner (linear system) solve and see whether
> the outer (Newton) iterations become more predictable.


> The other possibility is indeed in the communication "inexectness". Are
> there any asynchronous communications used in PETSc or in your code.
> Change all you can to synchronous communications and see whether results
> become more stable. 

Could you comment in particular on this latter?
Is there a way we can modify PETSc's behaviour via command line
options or otherwise?

Regards,
Aldo

-- 
Dr. Aldo Bonfiglioli
Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA)
Universita' della Basilicata
V.le dell'Ateneo lucano, 10 85100 Potenza ITALY
tel:+39.0971.205203 fax:+39.0971.205160

-------------- next part --------------
A non-text attachment was scrubbed...
Name: geometricPartitioning.pdf
Type: application/x-download
Size: 3634 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080311/214b5a29/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MetisPartitioning.pdf
Type: application/x-download
Size: 8193 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080311/214b5a29/attachment-0001.bin>