Non repeatability issue
Barry Smith
bsmith at mcs.anl.gov
Wed Mar 12 16:37:38 CDT 2008
Aldo,
Actually it is far easier than I thought. I have added the
argument -vecscatter_reproduce
that will cause the receives to always be processed in the same order
(though order or
operations in the MPI reductions may still result in slightly
different convergence histories.)
to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html
If you are using petsc-2.3.3 and do no wish to change to using
petsc-dev you can edit
src/vec/vec/utils/vpscat.h and replace the line
ierr = MPI_Waitany(nrecvs,rwaits,&imdex,&xrstatus);CHKERRQ(ierr);
with the two lines
imdex = count - 1;
ierr = MPI_Wait(rwaits+imdex,&xrstatus);CHKERRQ(ierr);
then run "make libfast" in that directory.
Hope this helps clear things up,
Barry
On Mar 12, 2008, at 11:02 AM, Aldo Bonfiglioli wrote:
> Barry,
> thanks for your prompt reply.
>
>> 1) Have you made runs where you require, say -ksp_rtol 1.e-12
>> to eliminate the effects of
>> not solving the linear systems accurately?
>
> I am trying this right now, although it may require very many linear
> iterations, particularly in the last Newton steps, when the CFL number
> becomes infinite.
> The systems arising from the fully coupled RANS eqns appear to be
> fairly ill-conditioned:
> even in the 2D cases we generally need to increase the level of fill
> beyond 0
> to obtain reasonable convergence of the linear solver.
>
> In the 3D case (internal flow through an elbow) the previous plots
> refer to, there are
> about 200,000 nodes times 5 dof per node and
> I have been using BJ+ILU(1) with GMRES(60).
>
>> 2) Have you run the exact example that you ran with geometric
>> decomposition also with
>> the parmetis decomposition? Is that what you sent? (This is too
>> eliminate any fundamental
>> differences with the two problems.)
>
> Precisely.
>
>> 3) In your plots you plot L_2 norm of the mass residual while
>> Newton is running on all
>> equations. This means the Newton's criteria for progress is based
>> on || u,v,m .....||
>> as it chugs along. What do plots of || u,v, m....|| (that is what
>> Newton calls the residual
>> when you use -snes_monitor) look like, are they also unstable? Are
>> they decreasing?
>
> All residuals appear to be fairly un-stable. This is shown in the
> enclosed plot.
> The strategy to ramp the CFL number looks at the ratio btw the
> current non-linear
> residual _of one of the conservation eqns_ and the initial residual.
> The Newton strategy adopted in the code is NOT implemented through
> PETSc's TS
> (just for "historical" reasons).
>
>> Sometimes people scale some equations stronger than others if
>> those are the residuals
>> they are most interested in pushing down. What happens if you
>> scale the mass residual
>> equations by some factor (say 100 or 1000) in your FormFunction?
>
> Never tried. We had once tested on a different problem row scaling
> without seeing noticeable differences. But I guess this may be
> different from what you mention.
>
>
> Regards,
> Aldo
>
> --
> Dr. Aldo Bonfiglioli
> Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA)
> Universita' della Basilicata
> V.le dell'Ateneo lucano, 10 85100 Potenza ITALY
> tel:+39.0971.205203 fax:+39.0971.205160
>
> <convhst.pdf>
More information about the petsc-users
mailing list