[petsc-users] Different results with different distribution of rows
Dave May
dave.mayhem23 at gmail.com
Tue Feb 9 08:06:32 CST 2016
If you don't specify preconditioner via -pc_type XXX, the default being
used is BJacobi-ILU.
This preconditioner will yield different results on different numbers of
MPI-processes, and will yield different results for a fixed number of
different MPI-processes, but with a different matrix partitioning. If you
operator is singular (or close to singular), ILU is likely to fail.
To be sure your code is working correctly, test it using a preconditioner
which isn't dependent on the partitioning of the matrix. I would use these
options:
-pc_type jacobi
-ksp_monitor_true_residual
The last option is useful as it will report both the preconditioned
residual and the true residual. If the operator is singular, or close to
singular, gmres-ILU or gmres-BJacobi-ILU can report a preconditioned
residual which is small, but is orders of magnitude different from the true
residual.
Thanks,
Dave
On 9 February 2016 at 14:39, Florian Lindner <mailinglists at xgm.de> wrote:
> Addition. The KSP Solver shows very different convergence:
>
>
> WRONG:
>
> [0] KSPConvergedDefault(): Linear solver has converged. Residual norm
> 6.832362172732e+06 is less than relative tolerance 1.000000000000e-09
> times initial right hand side norm 6.934533099989e+15 at iteration 8447
>
> RIGHT:
>
> [0] KSPConvergedDefault(): Linear solver has converged. Residual norm
> 7.959757133341e-08 is less than relative tolerance 1.000000000000e-09
> times initial right hand side norm 1.731788191624e+02 at iteration 9
>
> Best,
> Florian
>
> On Tue, 9 Feb 2016 14:06:01 +0100
> Florian Lindner <mailinglists at xgm.de> wrote:
>
> > Hello,
> >
> > I use PETSc with 4 MPI processes and I experience different results
> > when using different distribution of rows amoung ranks. The code looks
> > like that:
> >
> >
> > KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix);
> > // _solverRtol = 1e-9
> > KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT,
> > PETSC_DEFAULT);
> > KSPSetFromOptions(_solver);
> >
> > // means: MatGetVecs(matrix, nullptr, &vector);
> > petsc::Vector Au(_matrixC, "Au");
> > petsc::Vector out(_matrixC, "out");
> > petsc::Vector in(_matrixA, "in");
> >
> > // is an identity mapping here
> > ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping);
> >
> > // fill and assemble vector in
> >
> > in.view();
> > MatMultTranspose(_matrixA.matrix, in.vector, Au.vector);
> > Au.view();
> > KSPSolve(_solver, Au.vector, out.vector);
> >
> > out.view();
> >
> > I have experimented with two variants. The first one, non-working has
> > MatrixC rows distributed like that: 3, 4, 2, 2. The other, working,
> > one has 4, 3, 2, 2.
> >
> > All input values: _matrixA, _matrixC, in are identival, Au is
> > identical too, but out differs. See the results from object::view
> > below.
> >
> > Could the bad condition of matrixC be a problem? I'm not using any
> > special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ.
> >
> > Thanks,
> > Florian
> >
> > WRONG
> > =================
> >
> > Vec Object:in 4 MPI processes
> > type: mpi
> > Process [0]
> > 1
> > 2
> > Process [1]
> > 3
> > 4
> > Process [2]
> > 5
> > 6
> > Process [3]
> > 7
> > 8
> >
> > Vec Object:Au 4 MPI processes
> > type: mpi
> > Process [0]
> > 36
> > 74
> > 20
> > Process [1]
> > 1.09292
> > 2.09259
> > 3.18584
> > 4.20349
> > Process [2]
> > 5.29708
> > 6.31472
> > Process [3]
> > 7.24012
> > 8.23978
> >
> > // should not be result
> > Vec Object:out 4 MPI processes
> > type: mpi
> > Process [0]
> > -1.10633e+07
> > 618058
> > 9.01497e+06
> > Process [1]
> > 0.996195
> > 1.98711
> > 3.01111
> > 4.00203
> > Process [2]
> > 5.00736
> > 6.01644
> > Process [3]
> > 6.98534
> > 7.99442
> >
> > Mat Object:C 4 MPI processes
> > type: mpisbaij
> > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9,
> > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2)
> > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6,
> > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156)
> > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09)
> > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09)
> > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8,
> > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7,
> > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row
> > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8:
> > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10,
> > 0.0183156) row 10: (10, 1)
> >
> > Mat Object:A 4 MPI processes
> > type: mpiaij
> > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00
> > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07
> > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00
> > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02
> > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00
> > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00
> > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00
> > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02
> > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04
> > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07
> > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04
> > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00
> > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04
> > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00
> > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04
> > 1.83156e-02 1.83156e-02 1.00000e+00
> >
> >
> >
> > RIGHT
> > =================
> >
> > Vec Object:in 4 MPI processes
> > type: mpi
> > Process [0]
> > 1
> > 2
> > Process [1]
> > 3
> > 4
> > Process [2]
> > 5
> > 6
> > Process [3]
> > 7
> > 8
> >
> > Vec Object:Au 4 MPI processes
> > type: mpi
> > Process [0]
> > 36
> > 74
> > 20
> > 1.09292
> > Process [1]
> > 2.09259
> > 3.18584
> > 4.20349
> > Process [2]
> > 5.29708
> > 6.31472
> > Process [3]
> > 7.24012
> > 8.23978
> >
> > // should be result
> > Vec Object:out 4 MPI processes
> > type: mpi
> > Process [0]
> > 0
> > 0
> > 0
> > 1
> > Process [1]
> > 2
> > 3
> > 4
> > Process [2]
> > 5
> > 6
> > Process [3]
> > 7
> > 8
> >
> >
> > Mat Object:C 4 MPI processes
> > type: mpisbaij
> > row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9,
> > 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2)
> > (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6,
> > 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156)
> > (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09)
> > row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09)
> > (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8,
> > 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7,
> > 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row
> > 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8:
> > (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10,
> > 0.0183156) row 10: (10, 1)
> >
> > Mat Object:A 4 MPI processes
> > type: mpiaij
> > 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> > 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00
> > 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> > 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07
> > 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00
> > 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02
> > 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00
> > 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00
> > 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00
> > 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02
> > 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04
> > 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07
> > 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04
> > 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00
> > 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04
> > 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00
> > 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04
> > 1.83156e-02 1.83156e-02 1.00000e+00
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160209/9b1cef3b/attachment-0001.html>
More information about the petsc-users
mailing list