Dear all, I have recently observed a non-repeatibility issue with my unstructured CFD code. I solve the steady flow eqns. using pseudo-timestepping. PETSc is used to solve the large sparse linear systems arising at each psudo-time-step. Once close to the steady state solution, the time step size is increased to infinity so that Newton's method is recovered. Here below is a sample 2D calculation performed on 4 procs of a linux Beowulf cluster (but we also tested on a 4 procs ES40). The first two columns respectively show the time-step counter (i.e. the nonlinear iteration counter) and the KSP iterations required to solve the linear system. Cols. 5 - 8 display the nodal residuals (for mass, energy, etc.) In the early stages of the (non-linear) iterative process all residuals are identical among different runs. Once we get close to machine accuracy (iterations 48-50 in the example below), the residuals differ as do the KSP iterations for convergence. We believe to have identified the source of the differences in the following: At each step we solve a linear sistem of the form A x = rhs. Close to steady state, both x and rhs are close to machine eps. This is because rhs is the nodal residual which converges to machine eps as steady state is reached and x is the nodal increment (also converging to machine eps at steady state). As we look at the rhs (for instance) computed on each processor (that is the processor owned elements of a ghosted rhs Vec), this is identical between different runs. Once we call GhostUpdateBegin/End this is not any more true, although differences are of the order of machine eps. Our explanation is the following. The grid is partitioned so that the triangular cells belong only to a processor, but gridpoints (where the variables are stored) are shared among adjacent subdomains/procs. The Rhs is built in a FE manner by assembling contributions from surrounding cells. Given a gridpoint i shared by various subdomains (that is lying on an interdomain boundary), (*) Rhs_i = Rhs_i^p1 + Rhs_i^p2 + Rhs_i^p3 where the various contributions p1,p2, etc. are due to group of cells (triangles) belonging to different processors p1,p2,etc. At steady state Rhs_i must be of order machine eps, however some of the contributions will be of larger order of magnitude but with different sign (in order to cancel out) while other contributions could again be of order machine eps. E.g. Rhs_i^p1 = - Rhs_i^p2 and Rhs_i^p3 order machine eps . In this situation the result of the summation (*) will depend on the order by which the various terms on the right hand side are summed up. This is what we observe in practise on more than 2 procs.. When using two procs. there are only two contributions in the summation (*) and the convergence history is repeatable. Now, although this may look as a minor issue and in fact in the example below (inviscid flow past a 2D profile) as well as in many others all runs converge towards "steady state" we experienced the fact that on larger 3D simulations when we solve the RANS eqns using Newton's algorithm, the convergence history is run-dependent and different runs may behave fairly differently. An example is attached below: here we use ASM + ILU(1) + GMRES(60). Differences start to show up at the fourth non-linear iteration and run1 will eventually "converge" by reducing the non-linear residuals below 1.e-10, while the residuals of run2 stagnate. INVISCID 2D run Newton KSP +------- Non-linear residuals ------------+ step its run1 1 19 0.1516E+02 0.1642E+02 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1186E+01 0.2187E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.3674E+01 0.4492E+03 0.4005E-13 0.2490E-12 0.8549E-13 0.7615E-13 0.7203E+13 48 89 0.4874E+01 0.4560E+03 0.2639E-13 0.1710E-12 0.5629E-13 0.4947E-13 0.1093E+14 49 89 0.9156E+01 0.4671E+03 0.1841E-13 0.1334E-12 0.4205E-13 0.3460E-13 0.1567E+14 50 105 0.9106E+01 0.4780E+03 0.1485E-13 0.1138E-12 0.3245E-13 0.2475E-13 0.1942E+14 run2 1 19 0.6034E+01 0.7502E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.2107E+01 0.1126E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.1152E+02 0.4406E+03 0.3978E-13 0.2469E-12 0.8471E-13 0.7610E-13 0.7253E+13 48 89 0.5312E+01 0.4468E+03 0.2585E-13 0.1728E-12 0.5808E-13 0.5017E-13 0.1116E+14 49 91 0.7624E+01 0.4560E+03 0.1839E-13 0.1333E-12 0.4204E-13 0.3432E-13 0.1569E+14 50 106 0.5025E+01 0.4620E+03 0.1468E-13 0.1074E-12 0.3237E-13 0.2618E-13 0.1965E+14 run3 1 19 0.7433E+01 0.9449E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1460E+01 0.1495E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.9440E+01 0.4151E+03 0.3962E-13 0.2455E-12 0.8454E-13 0.7573E-13 0.7280E+13 48 89 0.1344E+02 0.4306E+03 0.2564E-13 0.1704E-12 0.5667E-13 0.5045E-13 0.1125E+14 49 94 0.8316E+01 0.4406E+03 0.1748E-13 0.1274E-12 0.4100E-13 0.3388E-13 0.1651E+14 50 104 0.4968E+01 0.4469E+03 0.1539E-13 0.1098E-12 0.3231E-13 0.2516E-13 0.1874E+14 run4 1 19 0.2243E+01 0.3578E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.4621E+01 0.2723E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.7702E+01 0.4093E+03 0.3982E-13 0.2487E-12 0.8516E-13 0.7588E-13 0.7244E+13 48 89 0.8366E+01 0.4188E+03 0.2588E-13 0.1739E-12 0.5760E-13 0.4989E-13 0.1114E+14 49 93 0.1078E+02 0.4315E+03 0.1808E-13 0.1248E-12 0.3912E-13 0.3438E-13 0.1596E+14 50 106 0.3265E+01 0.4363E+03 0.1549E-13 0.1135E-12 0.3467E-13 0.2506E-13 0.1863E+14 run5 1 19 0.8703E+01 0.1049E+02 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1402E+01 0.1512E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.9737E+01 0.4423E+03 0.3995E-13 0.2498E-12 0.8584E-13 0.7608E-13 0.7221E+13 48 89 0.3896E+01 0.4484E+03 0.2608E-13 0.1717E-12 0.5689E-13 0.4955E-13 0.1106E+14 49 90 0.2708E+01 0.4525E+03 0.1841E-13 0.1318E-12 0.4075E-13 0.3397E-13 0.1567E+14 50 104 0.8198E+01 0.4622E+03 0.1507E-13 0.1137E-12 0.3450E-13 0.2644E-13 0.1914E+14 3D RANS (on 14 procs.) +--------------- residuals -------------------------- + run1 1 300 0.5524E+02 0.7469E+02 0.3694E-07 0.5036E-07 0.2536E-07 0.6433E-07 0.3260E-06 0.5000E+02 2 8 0.2160E+01 0.9547E+02 0.2600E-06 0.3789E-07 0.1731E-06 0.2833E-06 0.3886E-03 0.7102E+01 3 10 0.2455E+01 0.1165E+03 0.1314E-06 0.7531E-07 0.1525E-06 0.2746E-06 0.7369E-04 0.1405E+02 4 16 0.3338E+01 0.1385E+03 0.9384E-07 0.4916E-07 0.1621E-06 0.1764E-06 0.4438E-04 0.1968E+02 5 18 0.3649E+01 0.1608E+03 0.3103E-07 0.3694E-07 0.1028E-06 0.9826E-07 0.2768E-04 0.5952E+02 ... 38 31 0.5803E+01 0.9777E+03 0.7378E-08 0.1099E-07 0.7864E-08 0.8981E-08 0.1958E-04 0.2503E+03 39 53 0.1009E+02 0.1006E+04 0.3250E-08 0.9071E-08 0.2718E-08 0.5060E-08 0.8113E-05 0.5683E+03 40 47 0.8849E+01 0.1034E+04 0.1351E-08 0.6064E-08 0.2013E-08 0.3336E-08 0.4884E-05 0.1367E+04 41 47 0.8856E+01 0.1061E+04 0.3722E-08 0.3737E-08 0.2168E-08 0.4815E-08 0.5811E-05 0.4963E+03 42 31 0.5822E+01 0.1086E+04 0.7197E-08 0.5195E-08 0.1029E-07 0.3716E-08 0.1042E-05 0.2566E+03 43 47 0.8854E+01 0.1113E+04 0.3824E-08 0.1100E-07 0.1788E-08 0.4469E-08 0.5128E-05 0.4830E+03 44 60 0.1162E+02 0.1143E+04 0.2248E-08 0.5162E-08 0.2033E-08 0.2756E-08 0.8732E-06 0.8216E+03 45 115 0.2117E+02 0.1183E+04 0.1570E-08 0.2492E-08 0.9909E-09 0.3213E-08 0.1739E-05 0.1176E+04 46 226 0.4071E+02 0.1243E+04 0.2806E-09 0.1260E-08 0.1742E-09 0.4544E-09 0.5404E-06 0.6582E+04 47 515 0.9198E+02 0.1353E+04 0.4936E-10 0.5056E-09 0.8438E-10 0.3446E-10 0.5972E-07 0.3742E+05 run2 1 300 0.5465E+02 0.7417E+02 0.3694E-07 0.5036E-07 0.2536E-07 0.6433E-07 0.3260E-06 0.5000E+02 2 8 0.2154E+01 0.9502E+02 0.2601E-06 0.3789E-07 0.1731E-06 0.2833E-06 0.3886E-03 0.7102E+01 3 10 0.2449E+01 0.1162E+03 0.1314E-06 0.7530E-07 0.1526E-06 0.2746E-06 0.7369E-04 0.1406E+02 4 18 0.3641E+01 0.1386E+03 0.9381E-07 0.4916E-07 0.1621E-06 0.1764E-06 0.4437E-04 0.1969E+02 5 18 0.3646E+01 0.1610E+03 0.3104E-07 0.3693E-07 0.1028E-06 0.9825E-07 0.2769E-04 0.5950E+02 ... 46 50 0.9383E+01 0.1100E+04 0.4251E-08 0.6184E-07 0.1024E-07 0.9866E-08 0.2737E-04 0.4344E+03 47 43 0.7984E+01 0.1127E+04 0.5129E-08 0.5265E-07 0.5989E-08 0.4442E-08 0.1972E-04 0.3601E+03 48 8 0.2164E+01 0.1148E+04 0.7553E-07 0.5328E-07 0.2634E-07 0.1430E-06 0.9913E-04 0.2445E+02 49 21 0.4108E+01 0.1171E+04 0.1561E-07 0.5173E-07 0.4150E-07 0.6183E-07 0.1828E-04 0.1183E+03 50 35 0.6471E+01 0.1196E+04 0.6973E-08 0.2463E-07 0.1185E-07 0.1466E-07 0.1397E-04 0.2649E+03 51 39 0.7233E+01 0.1222E+04 0.4899E-08 0.1581E-07 0.4077E-08 0.6338E-08 0.1131E-04 0.3770E+03 52 48 0.8969E+01 0.1249E+04 0.3529E-08 0.1495E-07 0.4077E-08 0.3640E-08 0.8881E-05 0.5234E+03 53 42 0.7789E+01 0.1276E+04 0.4898E-08 0.9435E-08 0.2343E-08 0.4393E-08 0.4517E-05 0.3771E+03 54 43 0.7984E+01 0.1303E+04 0.4259E-08 0.8999E-08 0.2747E-08 0.7004E-08 0.3261E-05 0.4337E+03 55 17 0.3496E+01 0.1325E+04 0.5685E-08 0.5167E-08 0.7471E-08 0.7705E-08 0.3686E-05 0.3249E+03