[petsc-users] Different results with different distribution of rows
Florian Lindner
mailinglists at xgm.de
Tue Feb 9 07:39:10 CST 2016
Addition. The KSP Solver shows very different convergence:
WRONG:
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm
6.832362172732e+06 is less than relative tolerance 1.000000000000e-09
times initial right hand side norm 6.934533099989e+15 at iteration 8447
RIGHT:
[0] KSPConvergedDefault(): Linear solver has converged. Residual norm
7.959757133341e-08 is less than relative tolerance 1.000000000000e-09
times initial right hand side norm 1.731788191624e+02 at iteration 9
Best,
Florian
On Tue, 9 Feb 2016 14:06:01 +0100
Florian Lindner <mailinglists at xgm.de> wrote:
> Hello,
>
> I use PETSc with 4 MPI processes and I experience different results
> when using different distribution of rows amoung ranks. The code looks
> like that:
>
>
> KSPSetOperators(_solver, _matrixC.matrix, _matrixC.matrix);
> // _solverRtol = 1e-9
> KSPSetTolerances(_solver, _solverRtol, PETSC_DEFAULT, PETSC_DEFAULT,
> PETSC_DEFAULT);
> KSPSetFromOptions(_solver);
>
> // means: MatGetVecs(matrix, nullptr, &vector);
> petsc::Vector Au(_matrixC, "Au");
> petsc::Vector out(_matrixC, "out");
> petsc::Vector in(_matrixA, "in");
>
> // is an identity mapping here
> ierr = VecSetLocalToGlobalMapping(in.vector, _ISmapping);
>
> // fill and assemble vector in
>
> in.view();
> MatMultTranspose(_matrixA.matrix, in.vector, Au.vector);
> Au.view();
> KSPSolve(_solver, Au.vector, out.vector);
>
> out.view();
>
> I have experimented with two variants. The first one, non-working has
> MatrixC rows distributed like that: 3, 4, 2, 2. The other, working,
> one has 4, 3, 2, 2.
>
> All input values: _matrixA, _matrixC, in are identival, Au is
> identical too, but out differs. See the results from object::view
> below.
>
> Could the bad condition of matrixC be a problem? I'm not using any
> special KSP options. matrixC is of type MATSBAIJ, matrixA is MATAIJ.
>
> Thanks,
> Florian
>
> WRONG
> =================
>
> Vec Object:in 4 MPI processes
> type: mpi
> Process [0]
> 1
> 2
> Process [1]
> 3
> 4
> Process [2]
> 5
> 6
> Process [3]
> 7
> 8
>
> Vec Object:Au 4 MPI processes
> type: mpi
> Process [0]
> 36
> 74
> 20
> Process [1]
> 1.09292
> 2.09259
> 3.18584
> 4.20349
> Process [2]
> 5.29708
> 6.31472
> Process [3]
> 7.24012
> 8.23978
>
> // should not be result
> Vec Object:out 4 MPI processes
> type: mpi
> Process [0]
> -1.10633e+07
> 618058
> 9.01497e+06
> Process [1]
> 0.996195
> 1.98711
> 3.01111
> 4.00203
> Process [2]
> 5.00736
> 6.01644
> Process [3]
> 6.98534
> 7.99442
>
> Mat Object:C 4 MPI processes
> type: mpisbaij
> row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9,
> 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2)
> (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6,
> 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156)
> (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09)
> row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09)
> (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8,
> 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7,
> 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row
> 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8:
> (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10,
> 0.0183156) row 10: (10, 1)
>
> Mat Object:A 4 MPI processes
> type: mpiaij
> 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00
> 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07
> 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00
> 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02
> 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00
> 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00
> 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00
> 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02
> 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04
> 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07
> 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04
> 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00
> 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04
> 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00
> 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04
> 1.83156e-02 1.83156e-02 1.00000e+00
>
>
>
> RIGHT
> =================
>
> Vec Object:in 4 MPI processes
> type: mpi
> Process [0]
> 1
> 2
> Process [1]
> 3
> 4
> Process [2]
> 5
> 6
> Process [3]
> 7
> 8
>
> Vec Object:Au 4 MPI processes
> type: mpi
> Process [0]
> 36
> 74
> 20
> 1.09292
> Process [1]
> 2.09259
> 3.18584
> 4.20349
> Process [2]
> 5.29708
> 6.31472
> Process [3]
> 7.24012
> 8.23978
>
> // should be result
> Vec Object:out 4 MPI processes
> type: mpi
> Process [0]
> 0
> 0
> 0
> 1
> Process [1]
> 2
> 3
> 4
> Process [2]
> 5
> 6
> Process [3]
> 7
> 8
>
>
> Mat Object:C 4 MPI processes
> type: mpisbaij
> row 0: (0, 0) (3, 1) (4, 1) (5, 1) (6, 1) (7, 1) (8, 1) (9,
> 1) (10, 1) row 1: (1, 0) (3, 0) (4, 0) (5, 1) (6, 1) (7, 2)
> (8, 2) (9, 3) (10, 3) row 2: (2, 0) (3, 0) (4, 1) (5, 0) (6,
> 1) (7, 0) (8, 1) (9, 0) (10, 1) row 3: (3, 1) (4, 0.0183156)
> (5, 0.0183156) (6, 0.000335463) (7, 1.12535e-07) (8, 2.06115e-09)
> row 4: (4, 1) (5, 0.000335463) (6, 0.0183156) (7, 2.06115e-09)
> (8, 1.12535e-07) row 5: (5, 1) (6, 0.0183156) (7, 0.0183156) (8,
> 0.000335463) (9, 1.12535e-07) (10, 2.06115e-09) row 6: (6, 1) (7,
> 0.000335463) (8, 0.0183156) (9, 2.06115e-09) (10, 1.12535e-07) row
> 7: (7, 1) (8, 0.0183156) (9, 0.0183156) (10, 0.000335463) row 8:
> (8, 1) (9, 0.000335463) (10, 0.0183156) row 9: (9, 1) (10,
> 0.0183156) row 10: (10, 1)
>
> Mat Object:A 4 MPI processes
> type: mpiaij
> 1.00000e+00 0.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> 1.83156e-02 3.35463e-04 1.12535e-07 2.06115e-09 0.00000e+00
> 0.00000e+00 1.00000e+00 0.00000e+00 1.00000e+00 1.83156e-02
> 1.00000e+00 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07
> 0.00000e+00 0.00000e+00 1.00000e+00 1.00000e+00 0.00000e+00
> 1.83156e-02 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02
> 3.35463e-04 1.12535e-07 2.06115e-09 1.00000e+00 1.00000e+00
> 1.00000e+00 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00
> 3.35463e-04 1.83156e-02 2.06115e-09 1.12535e-07 1.00000e+00
> 2.00000e+00 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02
> 3.35463e-04 1.00000e+00 1.83156e-02 1.83156e-02 3.35463e-04
> 1.00000e+00 2.00000e+00 1.00000e+00 2.06115e-09 1.12535e-07
> 3.35463e-04 1.83156e-02 1.83156e-02 1.00000e+00 3.35463e-04
> 1.83156e-02 1.00000e+00 3.00000e+00 0.00000e+00 0.00000e+00
> 0.00000e+00 1.12535e-07 2.06115e-09 1.83156e-02 3.35463e-04
> 1.00000e+00 1.83156e-02 1.00000e+00 3.00000e+00 1.00000e+00
> 0.00000e+00 0.00000e+00 2.06115e-09 1.12535e-07 3.35463e-04
> 1.83156e-02 1.83156e-02 1.00000e+00
More information about the petsc-users
mailing list