Non repeatability issue

[Barry Smith]

    Aldo,

      Actually it is far easier than I thought. I have added the  
argument -vecscatter_reproduce
that will cause the receives to always be processed in the same order  
(though order or
operations in the MPI reductions may still result in slightly  
different convergence histories.)
to petsc-dev http://www-unix.mcs.anl.gov/petsc/petsc-as/developers/index.html

    If you are using petsc-2.3.3 and do no wish to change to using  
petsc-dev you can edit
src/vec/vec/utils/vpscat.h and replace the line
	ierr = MPI_Waitany(nrecvs,rwaits,&imdex,&xrstatus);CHKERRQ(ierr);
with the two lines
	imdex = count - 1;
	ierr = MPI_Wait(rwaits+imdex,&xrstatus);CHKERRQ(ierr);
then run "make libfast" in that directory.

   Hope this helps clear things up,


    Barry

On Mar 12, 2008, at 11:02 AM, Aldo Bonfiglioli wrote:

> Barry,
> thanks for your prompt reply.
>
>> 1)    Have you made runs where you require, say -ksp_rtol 1.e-12  
>> to  eliminate the effects of
>> not solving the linear systems accurately?
>
> I am trying this right now, although it may require very many linear
> iterations, particularly in the last Newton steps, when the CFL number
> becomes infinite.
> The systems arising from the fully coupled RANS eqns appear to be  
> fairly ill-conditioned:
> even in the 2D cases we generally need to increase the level of fill  
> beyond 0
> to obtain reasonable convergence of the linear solver.
>
> In the 3D case (internal flow through an elbow) the previous plots  
> refer to, there are
> about 200,000 nodes times 5 dof per node and
> I have been using BJ+ILU(1) with GMRES(60).
>
>> 2) Have you run the exact example that you ran with geometric   
>> decomposition also with
>> the parmetis decomposition? Is that what you sent? (This is too   
>> eliminate any fundamental
>> differences with the two problems.)
>
> Precisely.
>
>> 3) In your plots you plot L_2 norm of the mass residual while  
>> Newton  is running on all
>> equations. This means the Newton's criteria for progress is based  
>> on  || u,v,m .....||
>> as it chugs along. What do plots of || u,v, m....|| (that is what   
>> Newton calls the residual
>> when you use -snes_monitor) look like, are they also unstable? Are   
>> they decreasing?
>
> All residuals appear to be fairly un-stable. This is shown in the  
> enclosed plot.
> The strategy to ramp the CFL number looks at the ratio btw the  
> current non-linear
> residual _of one of the conservation eqns_ and the initial residual.
> The Newton strategy adopted in the code is NOT implemented through  
> PETSc's TS
> (just for "historical" reasons).
>
>> Sometimes people scale some equations stronger than others if  
>> those  are the residuals
>> they are most interested in pushing down. What happens if you  
>> scale  the mass residual
>> equations by some factor (say 100 or 1000) in your FormFunction?
>
> Never tried. We had once tested on a different problem row scaling
> without seeing noticeable differences. But I guess this may be
> different from what you mention.
>
>
> Regards,
> Aldo
>
> -- 
> Dr. Aldo Bonfiglioli
> Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA)
> Universita' della Basilicata
> V.le dell'Ateneo lucano, 10 85100 Potenza ITALY
> tel:+39.0971.205203 fax:+39.0971.205160
>
> <convhst.pdf>