[petsc-users] Slower performance using more MPI processes

Junchao Zhang junchao.zhang at gmail.com
Fri Sep 8 17:38:54 CDT 2023


Hi, Chris,
  this is my result of running your test on a local machine (8 cores, 16
hardware threads).  I use OpenMPI with command line like
    mpirun --bind-to core -n <np> ./main
np=1 Solve duration: 2132
np=2 Solve duration: 1116
np=4 Solve duration: 990
np=8 Solve duration: 1257

Note when I used
$ mpirun  -n 4 ./main
Solve duration: 22693

It suggests binding is important.

--Junchao Zhang


On Fri, Sep 8, 2023 at 4:53 PM Barry Smith <bsmith at petsc.dev> wrote:

>
> For the size problem you are running this is very unexpected.  The
> reductions should only start to dominate for thousands of MPI ranks, not 2.
>
> First thing I recommend is to run the streams benchmark. Then check the
> binding that MPI is doing for the two processes. You want to bind to cores
> in different NUMA regions. It could be that it is binding both processes to
> cores that share the same cache. The MatSolve should be embarressingly
> parallel but you are getting almost no speed up in it, so something very
> "wrong" is happening.
>
>
>
> Event                Count      Time (sec)     Flop
>        --- Global ---  --- Stage ----  Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen
>  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
> VecDot               182 1.0 1.9998e-01 1.0 2.26e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  7  5  0  0  0   7  5  0  0  0  1129
> VecDotNorm2           91 1.0 6.6214e-02 1.0 2.26e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  2  5  0  0  0   2  5  0  0  0  3409
> VecNorm               92 1.0 1.4790e-01 1.0 1.14e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  5  3  0  0  0   5  3  0  0  0   771
>
> VecDot               198 1.0 2.1037e+00 1.1 1.23e+08 1.0 0.0e+00 0.0e+00
> 2.0e+02 33  5  0  0 43  33  5  0  0 45   117
> VecDotNorm2           99 1.0 5.0169e-01 1.2 1.23e+08 1.0 0.0e+00 0.0e+00
> 9.9e+01  7  5  0  0 22   7  5  0  0 22   489
> VecNorm              100 1.0 1.3131e+00 1.0 6.20e+07 1.0 0.0e+00 0.0e+00
> 1.0e+02 21  3  0  0 22  21  3  0  0 23    94
> VecScatterEnd        198 1.0 7.6160e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 10  0  0  0  0  10  0  0  0  0     0
>
> MatSolve             183 1.0 1.0882e+00 1.0 1.43e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00 40 35  0  0  0  40 35  0  0  0  1318
> MatSolve             199 1.0 8.9666e-01 1.2 7.75e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00 13 35  0  0  0  13 35  0  0  0  1729
>
>
> On Sep 8, 2023, at 5:22 PM, Chris Hewson <chris at resfrac.com> wrote:
>
> Thanks for the quick response.
>
> The links to the log view files are below:
> 2 ranks:
>
> https://drive.google.com/file/d/1PGRsiHypWtN5h3uxdJBKy9WzEkE0mUgO/view?usp=drive_link
>
> 1 rank:
>
> https://drive.google.com/file/d/1hB2XyoNtLMHseZUT7jCuiixTQeBi_tjJ/view?usp=drive_link
>
> I'll also attach them to this email:
> **************************** 1 RANK ******************************
> ------------------------------------------------------------------ PETSc
> Performance Summary:
> ------------------------------------------------------------------
>
> ./petsc-testing on a  named ubuntu-office with 1 processor, by chewson Fri
> Sep  8 15:16:51 2023
> Using Petsc Release Version 3.19.5, unknown
>
>                          Max       Max/Min     Avg       Total
> Time (sec):           2.746e+00     1.000   2.746e+00
> Objects:              2.100e+01     1.000   2.100e+01
> Flops:                4.117e+09     1.000   4.117e+09  4.117e+09
> Flops/sec:            1.499e+09     1.000   1.499e+09  1.499e+09
> MPI Msg Count:        0.000e+00     0.000   0.000e+00  0.000e+00
> MPI Msg Len (bytes):  0.000e+00     0.000   0.000e+00  0.000e+00
> MPI Reductions:       0.000e+00     0.000
>
> Flop counting convention: 1 flop = 1 real number operation of type
> (multiply/divide/add/subtract)
>                             e.g., VecAXPY() for real vectors of length N
> --> 2N flops
>                             and VecAXPY() for complex vectors of length N
> --> 8N flops
>
> Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages
> ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total    Count
> %Total     Avg         %Total    Count   %Total
>  0:      Main Stage: 2.7458e+00 100.0%  4.1167e+09 100.0%  0.000e+00
> 0.0%  0.000e+00        0.0%  0.000e+00   0.0%
>
>
> ------------------------------------------------------------------------------------------------------------------------
> See the 'Profiling' chapter of the users' manual for details on
> interpreting output.
> Phase summary info:
>    Count: number of times phase was executed
>    Time and Flop: Max - maximum over all processors
>                   Ratio - ratio of maximum to minimum over all processors
>    Mess: number of messages sent
>    AvgLen: average message length (bytes)
>    Reduct: number of global reductions
>    Global: entire computation
>    Stage: stages of a computation. Set stages with PetscLogStagePush() and
> PetscLogStagePop().
>       %T - percent time in this phase         %F - percent flop in this
> phase
>       %M - percent messages in this phase     %L - percent message lengths
> in this phase
>       %R - percent reductions in this phase
>    Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over
> all processors)
>
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop
>        --- Global ---  --- Stage ----  Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen
>  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
> ------------------------------------------------------------------------------------------------------------------------
>
> --- Event Stage 0: Main Stage
>
> MatMult              182 1.0 8.0351e-01 1.0 1.43e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00 29 35  0  0  0  29 35  0  0  0  1775
> MatSolve             183 1.0 1.0882e+00 1.0 1.43e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00 40 35  0  0  0  40 35  0  0  0  1318
> MatLUFactorNum         1 1.0 1.3892e-02 1.0 1.30e+07 1.0 0.0e+00 0.0e+00
> 0.0e+00  1  0  0  0  0   1  0  0  0  0   934
> MatILUFactorSym        1 1.0 2.1567e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> MatAssemblyBegin       1 1.0 1.0420e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatAssemblyEnd         1 1.0 6.9049e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetRowIJ            1 1.0 3.8500e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetOrdering         1 1.0 1.7026e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatLoad                1 1.0 6.6749e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  2  0  0  0  0   2  0  0  0  0     0
> VecDot               182 1.0 1.9998e-01 1.0 2.26e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  7  5  0  0  0   7  5  0  0  0  1129
> VecDotNorm2           91 1.0 6.6214e-02 1.0 2.26e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  2  5  0  0  0   2  5  0  0  0  3409
> VecNorm               92 1.0 1.4790e-01 1.0 1.14e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  5  3  0  0  0   5  3  0  0  0   771
> VecCopy                2 1.0 6.8473e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet                 3 1.0 1.3256e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecAXPBYCZ           182 1.0 1.6542e-01 1.0 4.51e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  6 11  0  0  0   6 11  0  0  0  2729
> VecWAXPY             182 1.0 1.4476e-01 1.0 2.26e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  5  5  0  0  0   5  5  0  0  0  1559
> VecLoad                2 1.0 1.0104e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSetUp               1 1.0 9.9204e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 2.6210e+00 1.0 4.10e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00 95 100  0  0  0  95 100  0  0  0  1566
> PCSetUp                1 1.0 3.7232e-02 1.0 1.30e+07 1.0 0.0e+00 0.0e+00
> 0.0e+00  1  0  0  0  0   1  0  0  0  0   349
> PCApply              183 1.0 1.0885e+00 1.0 1.43e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00 40 35  0  0  0  40 35  0  0  0  1318
>
> --- Event Stage 1: Unknown
>
>
> ------------------------------------------------------------------------------------------------------------------------
>
> Object Type          Creations   Destructions. Reports information only
> for process 0.
>
> --- Event Stage 0: Main Stage
>
>               Viewer     4              1
>               Matrix     3              1
>               Vector     9              1
>        Krylov Solver     1              0
>       Preconditioner     1              0
>            Index Set     3              0
>
> --- Event Stage 1: Unknown
>
>
> ========================================================================================================================
> Average time to get PetscTime(): 1.51e-08
> #PETSc Option Table entries:
> -log_view # (source: command line)
> #End of PETSc Option Table entries
> Compiled without FORTRAN kernels
> Compiled with full precision matrices (default)
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
> sizeof(PetscScalar) 8 sizeof(PetscInt) 4
> Configure options: --with-debugging=0 --prefix=/opt/anl/petsc-3.19.5
> --download-mumps --download-scalapack --with-mpi=1
> --with-mpi-dir=/opt/anl/mpich COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2
> -----------------------------------------
> Libraries compiled on 2023-09-08 16:27:49 on ubuntu-office
> Machine characteristics: Linux-6.2.0-26-generic-x86_64-with-glibc2.35
> Using PETSc directory: /opt/anl/petsc-3.19.5
> Using PETSc arch:
> -----------------------------------------
>
> Using C compiler: /opt/anl/mpich/bin/mpicc  -fPIC -Wall -Wwrite-strings
> -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow
> -fstack-protector -fvisibility=hidden -O2
> Using Fortran compiler: /opt/anl/mpich/bin/mpif90  -fPIC -Wall
> -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch
> -Wno-unused-dummy-argument -O2
> -----------------------------------------
>
> Using include paths: -I/opt/anl/petsc-3.19.5/include
> -I/opt/anl/mpich/include
> -----------------------------------------
>
> Using C linker: /opt/anl/mpich/bin/mpicc
> Using Fortran linker: /opt/anl/mpich/bin/mpif90
> Using libraries: -Wl,-rpath,/opt/anl/petsc-3.19.5/lib
> -L/opt/anl/petsc-3.19.5/lib -lpetsc -Wl,-rpath,/opt/anl/petsc-3.19.5/lib
> -L/opt/anl/petsc-3.19.5/lib -Wl,-rpath,/opt/anl/mpich/lib
> -L/opt/anl/mpich/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11
> -L/usr/lib/gcc/x86_64-linux-gnu/11 -ldmumps -lmumps_common -lpord -lpthread
> -lscalapack -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm
> -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath
> -----------------------------------------
>
> ************************* 2 RANKS ************************************
> ------------------------------------------------------------------ PETSc
> Performance Summary:
> ------------------------------------------------------------------
>
> ./petsc-testing on a  named ubuntu-office with 2 processors, by chewson
> Fri Sep  8 15:16:43 2023
> Using Petsc Release Version 3.19.5, unknown
>
>                          Max       Max/Min     Avg       Total
> Time (sec):           6.167e+00     1.001   6.164e+00
> Objects:              3.200e+01     1.000   3.200e+01
> Flops:                2.233e+09     1.000   2.233e+09  4.467e+09
> Flops/sec:            3.625e+08     1.001   3.623e+08  7.247e+08
> MPI Msg Count:        2.050e+02     1.000   2.050e+02  4.100e+02
> MPI Msg Len (bytes):  3.437e+07     1.000   1.676e+05  6.874e+07
> MPI Reductions:       4.580e+02     1.000
>
> Flop counting convention: 1 flop = 1 real number operation of type
> (multiply/divide/add/subtract)
>                             e.g., VecAXPY() for real vectors of length N
> --> 2N flops
>                             and VecAXPY() for complex vectors of length N
> --> 8N flops
>
> Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages
> ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total    Count
> %Total     Avg         %Total    Count   %Total
>  0:      Main Stage: 6.1642e+00 100.0%  4.4670e+09 100.0%  4.100e+02
> 100.0%  1.676e+05      100.0%  4.400e+02  96.1%
>
>
> ------------------------------------------------------------------------------------------------------------------------
> See the 'Profiling' chapter of the users' manual for details on
> interpreting output.
> Phase summary info:
>    Count: number of times phase was executed
>    Time and Flop: Max - maximum over all processors
>                   Ratio - ratio of maximum to minimum over all processors
>    Mess: number of messages sent
>    AvgLen: average message length (bytes)
>    Reduct: number of global reductions
>    Global: entire computation
>    Stage: stages of a computation. Set stages with PetscLogStagePush() and
> PetscLogStagePop().
>       %T - percent time in this phase         %F - percent flop in this
> phase
>       %M - percent messages in this phase     %L - percent message lengths
> in this phase
>       %R - percent reductions in this phase
>    Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over
> all processors)
>
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop
>        --- Global ---  --- Stage ----  Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen
>  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
> ------------------------------------------------------------------------------------------------------------------------
>
> --- Event Stage 0: Main Stage
>
> BuildTwoSided          1 1.0 3.1824e-05 1.0 0.00e+00 0.0 2.0e+00 4.0e+00
> 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMult              198 1.0 1.3480e+00 1.4 7.76e+08 1.0 4.0e+02 9.4e+04
> 0.0e+00 19 35 97 54  0  19 35 97 54  0  1151
> MatSolve             199 1.0 8.9666e-01 1.2 7.75e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00 13 35  0  0  0  13 35  0  0  0  1729
> MatLUFactorNum         1 1.0 7.1852e-03 1.0 6.43e+06 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  1789
> MatILUFactorSym        1 1.0 1.0472e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatAssemblyBegin       1 1.0 9.8700e-07 1.4 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatAssemblyEnd         1 1.0 6.8341e-03 1.1 0.00e+00 0.0 4.0e+00 2.3e+04
> 5.0e+00  0  0  1  0  1   0  0  1  0  1     0
> MatGetRowIJ            1 1.0 1.9930e-06 6.3 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetOrdering         1 1.0 7.4472e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatLoad                1 1.0 9.8562e-02 1.0 0.00e+00 0.0 1.0e+01 2.7e+06
> 1.7e+01  2  0  2 39  4   2  0  2 39  4     0
> VecDot               198 1.0 2.1037e+00 1.1 1.23e+08 1.0 0.0e+00 0.0e+00
> 2.0e+02 33  5  0  0 43  33  5  0  0 45   117
> VecDotNorm2           99 1.0 5.0169e-01 1.2 1.23e+08 1.0 0.0e+00 0.0e+00
> 9.9e+01  7  5  0  0 22   7  5  0  0 22   489
> VecNorm              100 1.0 1.3131e+00 1.0 6.20e+07 1.0 0.0e+00 0.0e+00
> 1.0e+02 21  3  0  0 22  21  3  0  0 23    94
> VecCopy                2 1.0 7.4971e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet               202 1.0 8.0035e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> VecAXPBYCZ           198 1.0 1.2889e-01 1.5 2.46e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  2 11  0  0  0   2 11  0  0  0  3811
> VecWAXPY             198 1.0 9.1526e-02 1.0 1.23e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  1  5  0  0  0   1  5  0  0  0  2683
> VecLoad                2 1.0 9.8983e-03 1.0 0.00e+00 0.0 4.0e+00 1.2e+06
> 1.6e+01  0  0  1  7  3   0  0  1  7  4     0
> VecScatterBegin      198 1.0 1.2941e-03 1.0 0.00e+00 0.0 4.0e+02 9.4e+04
> 0.0e+00  0  0 97 54  0   0  0 97 54  0     0
> VecScatterEnd        198 1.0 7.6160e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 10  0  0  0  0  10  0  0  0  0     0
> SFSetGraph             1 1.0 7.6630e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFSetUp                1 1.0 1.2410e-04 1.0 0.00e+00 0.0 4.0e+00 2.3e+04
> 1.0e+00  0  0  1  0  0   0  0  1  0  0     0
> SFPack               198 1.0 5.1814e-05 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFUnpack             198 1.0 3.8273e-05 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSetUp               2 1.0 4.7077e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
> 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 6.0344e+00 1.0 2.23e+09 1.0 4.0e+02 9.4e+04
> 4.0e+02 98 100 97 54 87  98 100 97 54 90   738
> PCSetUp                2 1.0 1.8496e-02 1.0 6.43e+06 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0   695
> PCSetUpOnBlocks        1 1.0 1.8435e-02 1.0 6.43e+06 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0   697
> PCApply              199 1.0 9.5103e-01 1.1 7.75e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00 15 35  0  0  0  15 35  0  0  0  1630
>
> --- Event Stage 1: Unknown
>
>
> ------------------------------------------------------------------------------------------------------------------------
>
> Object Type          Creations   Destructions. Reports information only
> for process 0.
>
> --- Event Stage 0: Main Stage
>
>               Viewer     4              1
>               Matrix     5              1
>               Vector    13              2
>            Index Set     5              2
>    Star Forest Graph     1              0
>        Krylov Solver     2              0
>       Preconditioner     2              0
>
> --- Event Stage 1: Unknown
>
>
> ========================================================================================================================
> Average time to get PetscTime(): 2.47e-08
> Average time for MPI_Barrier(): 4.406e-07
> Average time for zero size MPI_Send(): 4.769e-06
> #PETSc Option Table entries:
> -log_view # (source: command line)
> #End of PETSc Option Table entries
> Compiled without FORTRAN kernels
> Compiled with full precision matrices (default)
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
> sizeof(PetscScalar) 8 sizeof(PetscInt) 4
> Configure options: --with-debugging=0 --prefix=/opt/anl/petsc-3.19.5
> --download-mumps --download-scalapack --with-mpi=1
> --with-mpi-dir=/opt/anl/mpich COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2
> -----------------------------------------
> Libraries compiled on 2023-09-08 16:27:49 on ubuntu-office
> Machine characteristics: Linux-6.2.0-26-generic-x86_64-with-glibc2.35
> Using PETSc directory: /opt/anl/petsc-3.19.5
> Using PETSc arch:
> -----------------------------------------
>
> Using C compiler: /opt/anl/mpich/bin/mpicc  -fPIC -Wall -Wwrite-strings
> -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow
> -fstack-protector -fvisibility=hidden -O2
> Using Fortran compiler: /opt/anl/mpich/bin/mpif90  -fPIC -Wall
> -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch
> -Wno-unused-dummy-argument -O2
> -----------------------------------------
>
> Using include paths: -I/opt/anl/petsc-3.19.5/include
> -I/opt/anl/mpich/include
> -----------------------------------------
>
> Using C linker: /opt/anl/mpich/bin/mpicc
> Using Fortran linker: /opt/anl/mpich/bin/mpif90
> Using libraries: -Wl,-rpath,/opt/anl/petsc-3.19.5/lib
> -L/opt/anl/petsc-3.19.5/lib -lpetsc -Wl,-rpath,/opt/anl/petsc-3.19.5/lib
> -L/opt/anl/petsc-3.19.5/lib -Wl,-rpath,/opt/anl/mpich/lib
> -L/opt/anl/mpich/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/11
> -L/usr/lib/gcc/x86_64-linux-gnu/11 -ldmumps -lmumps_common -lpord -lpthread
> -lscalapack -llapack -lblas -lm -lX11 -lmpifort -lmpi -lgfortran -lm
> -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -lquadmath
> -----------------------------------------
>
> Chris
>
>
> On Fri, Sep 8, 2023 at 3:00 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>>   It would be very helpful if you could run on 1 and 2 ranks with
>> -log_view and send all the output.
>>
>>
>>
>> On Sep 8, 2023, at 4:52 PM, Chris Hewson <chris at resfrac.com> wrote:
>>
>> Hi There,
>>
>> I am trying to solve a linear problem and am having an issue when I use
>> more MPI processes with the KSPsolve slowing down considerably the more
>> processes I add.
>>
>> The matrix itself is 620100 X 620100 with ~5 million non-zero entries, I
>> am using petsc version 3.19.5 and have tried with a couple different
>> versions of mpich getting the same behavior (v4.1.2 w/ device ch4:ofi and
>> v3.3.2 w/ ch3:sock).
>>
>> In testing, I've noticed the following trend for speed for the KSPSolve
>> function call:
>> 1 core: 4042 ms
>> 2 core: 7085 ms
>> 4 core: 26573 ms
>> 8 core: 65745 ms
>> 16 core: 149283 ms
>>
>> This was all done on a single node machine w/ 16 non-hyperthreaded cores.
>> We solve quite a few different matrices with PETSc using MPI and haven't
>> noticed an impact like this on performance before.
>>
>> I am very confused by this and am a little stumped at the moment as to
>> why this was happening. I've been using the KSPBCGS solver to solve the
>> problem. I have tried with multiple different solvers and pre-conditioners
>> (we usually don't use a pre-conditioner for this part of our code).
>>
>> It did seem that using the piped BCGS solver did help improve the
>> parallel speed slightly (maybe 15%), but it still doesn't come close to the
>> single threaded speed.
>>
>> I'll attach a link to a folder that contains the specific A, x and b
>> matrices for this problem, as well as a main.cpp file that I was using for
>> testing.
>>
>>
>> https://drive.google.com/drive/folders/1CEDinKxu8ZbKpLtwmqKqP1ZIDG7JvDI1?usp=sharing
>>
>> I was testing this in our main code base, but don't include that here,
>> and observe very similar speed results to the ones above. We do use Metis
>> to graph partition in our own code and checked the vector and matrix
>> partitioning and that all made sense. I could be doing the partitioning
>> incorrectly in the example (not 100% sure how it works with the viewer/load
>> functions).
>>
>> Any insight or thoughts on this would be greatly appreciated.
>>
>> Thanks,
>>
>> *Chris Hewson*
>> Senior Reservoir Simulation Engineer
>> ResFrac
>> +1.587.575.9792
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230908/ef518a70/attachment-0001.html>


More information about the petsc-users mailing list