[petsc-users] Running CG with HYPRE AMG preconditioner in AMD GPUs

Mark Adams mfadams at lbl.gov
Tue Mar 19 15:57:53 CDT 2024


[keep on list]

I have little experience with running hypre on GPUs but others might have
more.

1M dogs/node is not a lot and NVIDIA has larger L1 cache and more mature
compilers, etc. so it is not surprising that NVIDIA is faster.
I suspect the gap would narrow with a larger problem.

Also, why are you using Kokkos? It should not make a difference but you
could check easily. Just use -vec_type hip with your current code.

You could also test with GAMG, -pc_type gamg

Mark


On Tue, Mar 19, 2024 at 4:12 PM Vanella, Marcos (Fed) <
marcos.vanella at nist.gov> wrote:

> Hi Mark, I run a canonical test we have to time our code. It is a propane
> fire on a burner within a box with around 1 million cells.
> I split the problem in 4 GPUS, single node, both in Polaris and Frontier.
> I compiled PETSc with gnu and HYPRE being downloaded and the following
> configure options:
>
>
>    - Polaris:
>    $./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3"
>    FCOPTFLAGS="-O3" CUDAOPTFLAGS="-O3" --with-debugging=0
>    --download-suitesparse --download-hypre --with-cuda --with-cc=cc
>    --with-cxx=CC --with-fc=ftn --with-cudac=nvcc --with-cuda-arch=80
>    --download-cmake
>
>
>
>    - Frontier:
>    $./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3"
>    FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0 --with-cc=cc
>    --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc
>    --LIBS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a}
>    ${PE_MPICH_GTL_LIBS_amd_gfx90a}" --download-kokkos
>    --download-kokkos-kernels --download-suitesparse --download-hypre
>    --download-cmake
>
>
> Our code was compiled also with gnu compilers and -O3 flag. I used latest
> (from this week) PETSc repo update. These are the timings for the test case:
>
>
>    - 8 meshes + 1Million cells case, 8 MPI processes, 4 GPUS, 2 MPI Procs
>    per GPU, 1 sec run time (~580 time steps, ~1160 Poisson solves):
>
>
> System              Poisson Solver          GPU Implementation
> Poisson Wall time (sec)         Total Wall time (sec)
> Polaris             CG + HYPRE PC           CUDA
> 80                              287
> Frontier            CG + HYPRE PC           Kokkos + HIP
> 158                             401
>
> It is interesting to see that the Poisson solves take twice the time in
> Frontier than in Polaris.
> Do you have experience on running HYPRE AMG on these machines? Is this
> difference between the CUDA implementation and Kokkos-kernels to be
> expected?
>
> I can run the case in both computers with the log flags you suggest. Might
> give more information on where the differences are.
>
> Thank you for your time,
> Marcos
>
>
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Tuesday, March 5, 2024 2:41 PM
> *To:* Vanella, Marcos (Fed) <marcos.vanella at nist.gov>
> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Running CG with HYPRE AMG preconditioner in
> AMD GPUs
>
> You can run with -log_view_gpu_time to get rid of the nans and get more
> data.
>
> You can run with -ksp_view to get more info on the solver and send that
> output.
>
> -options_left is also good to use so we can see what parameters you used.
>
> The last 100 in this row:
>
> KSPSolve            1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04
> 3.1e+04 12 100 100 100 49  12 100 100 100 98  2503    -nan      0 1.80e-05
>    0 0.00e+00  100
>
> tells us that all the flops were logged on GPUs.
>
> You do need at least 100K equations per GPU to see speedup, so don't worry
> about small problems.
>
> Mark
>
>
>
>
> On Tue, Mar 5, 2024 at 12:52 PM Vanella, Marcos (Fed) via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos
> and hip options: ./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3"
> FOPTFLAGS="-O3" FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hi all, I compiled the latest PETSc source in Frontier using gcc+kokkos
> and hip options:
>
> ./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3"
> FCOPTFLAGS="-O3" HIPOPTFLAGS="-O3" --with-debugging=0 --with-cc=cc
> --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc
> --LIBS="-L${MPICH_DIR}/lib -lmpi ${PE_MPICH_GTL_DIR_amd_gfx90a}
> ${PE_MPICH_GTL_LIBS_amd_gfx90a}" --download-kokkos
> --download-kokkos-kernels --download-suitesparse --download-hypre
> --download-cmake
>
> and have started testing our code solving a Poisson linear system with CG
> + HYPRE preconditioner. Timings look rather high compared to compilations
> done on other machines that have NVIDIA cards. They are also not changing
> when using more than one GPU for the simple test I doing.
> Does anyone happen to know if HYPRE has an hip GPU implementation for
> Boomer AMG and is it compiled when configuring PETSc?
>
> Thanks!
>
> Marcos
>
>
> PS: This is what I see on the log file (-log_view) when running the case
> with 2 GPUs in the node:
>
>
> ------------------------------------------------------------------ PETSc
> Performance Summary:
> ------------------------------------------------------------------
>
> /ccs/home/vanellam/Firemodels_fork/fds/Build/mpich_gnu_frontier/fds_mpich_gnu_frontier
> on a arch-linux-frontier-opt-gcc named frontier04119 with 4 processors, by
> vanellam Tue Mar  5 12:42:29 2024
> Using Petsc Development GIT revision: v3.20.5-713-gabdf6bc0fcf  GIT Date:
> 2024-03-05 01:04:54 +0000
>
>                          Max       Max/Min     Avg       Total
> Time (sec):           8.368e+02     1.000   8.368e+02
> Objects:              0.000e+00     0.000   0.000e+00
> Flops:                2.546e+11     0.000   1.270e+11  5.079e+11
> Flops/sec:            3.043e+08     0.000   1.518e+08  6.070e+08
> MPI Msg Count:        1.950e+04     0.000   9.748e+03  3.899e+04
> MPI Msg Len (bytes):  1.560e+09     0.000   7.999e+04  3.119e+09
> MPI Reductions:       6.331e+04   2877.545
>
> Flop counting convention: 1 flop = 1 real number operation of type
> (multiply/divide/add/subtract)
>                             e.g., VecAXPY() for real vectors of length N
> --> 2N flops
>                             and VecAXPY() for complex vectors of length N
> --> 8N flops
>
> Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages
> ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total    Count
> %Total     Avg         %Total    Count   %Total
>  0:      Main Stage: 8.3676e+02 100.0%  5.0792e+11 100.0%  3.899e+04
> 100.0%  7.999e+04      100.0%  3.164e+04  50.0%
>
>
> ------------------------------------------------------------------------------------------------------------------------
> See the 'Profiling' chapter of the users' manual for details on
> interpreting output.
> Phase summary info:
>    Count: number of times phase was executed
>    Time and Flop: Max - maximum over all processors
>                   Ratio - ratio of maximum to minimum over all processors
>    Mess: number of messages sent
>    AvgLen: average message length (bytes)
>    Reduct: number of global reductions
>    Global: entire computation
>    Stage: stages of a computation. Set stages with PetscLogStagePush() and
> PetscLogStagePop().
>       %T - percent time in this phase         %F - percent flop in this
> phase
>       %M - percent messages in this phase     %L - percent message lengths
> in this phase
>       %R - percent reductions in this phase
>    Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over
> all processors)
>    GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU
> time over all processors)
>    CpuToGpu Count: total number of CPU to GPU copies per processor
>    CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per
> processor)
>    GpuToCpu Count: total number of GPU to CPU copies per processor
>    GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per
> processor)
>    GPU %F: percent flops on GPU in this event
>
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop
>        --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   -
> GpuToCpu - GPU
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen
>  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size
> Count   Size  %F
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> --- Event Stage 0: Main Stage
>
> BuildTwoSided       1201 0.0   nan nan 0.00e+00 0.0 2.0e+00 4.0e+00
> 6.0e+02  0  0  0  0  1   0  0  0  0  2  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> BuildTwoSidedF      1200 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 6.0e+02  0  0  0  0  1   0  0  0  0  2  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> MatMult            19494 0.0   nan nan 1.35e+11 0.0 3.9e+04 8.0e+04
> 0.0e+00  7 53 100 100  0   7 53 100 100  0  -nan    -nan      0 1.80e-05
>  0 0.00e+00  100
> MatConvert             3 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 1.5e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> MatAssemblyBegin       2 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 1.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> MatAssemblyEnd         2 0.0   nan nan 0.00e+00 0.0 4.0e+00 2.0e+04
> 3.5e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> VecTDot            41382 0.0   nan nan 4.14e+10 0.0 0.0e+00 0.0e+00
> 2.1e+04  0 16  0  0 33   0 16  0  0 65  -nan    -nan      0 0.00e+00    0
> 0.00e+00  100
> VecNorm            20691 0.0   nan nan 2.07e+10 0.0 0.0e+00 0.0e+00
> 1.0e+04  0  8  0  0 16   0  8  0  0 33  -nan    -nan      0 0.00e+00    0
> 0.00e+00  100
> VecCopy             2394 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> VecSet             21888 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> VecAXPY            38988 0.0   nan nan 3.90e+10 0.0 0.0e+00 0.0e+00
> 0.0e+00  0 15  0  0  0   0 15  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  100
> VecAYPX            18297 0.0   nan nan 1.83e+10 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  7  0  0  0   0  7  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  100
> VecAssemblyBegin    1197 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 6.0e+02  0  0  0  0  1   0  0  0  0  2  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> VecAssemblyEnd      1197 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> VecScatterBegin    19494 0.0   nan nan 0.00e+00 0.0 3.9e+04 8.0e+04
> 0.0e+00  0  0 100 100  0   0  0 100 100  0  -nan    -nan      0 1.80e-05
>  0 0.00e+00  0
> VecScatterEnd      19494 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> SFSetGraph             1 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> SFSetUp                1 0.0   nan nan 0.00e+00 0.0 4.0e+00 2.0e+04
> 5.0e-01  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> SFPack             19494 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 1.80e-05    0
> 0.00e+00  0
> SFUnpack           19494 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> KSPSetUp               1 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> KSPSolve            1197 0.0 2.0291e+02 0.0 2.55e+11 0.0 3.9e+04 8.0e+04
> 3.1e+04 12 100 100 100 49  12 100 100 100 98  2503    -nan      0 1.80e-05
>    0 0.00e+00  100
> PCSetUp                1 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 1.5e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
> PCApply            20691 0.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  5  0  0  0  0   5  0  0  0  0  -nan    -nan      0 0.00e+00    0
> 0.00e+00  0
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Object Type          Creations   Destructions. Reports information only
> for process 0.
>
> --- Event Stage 0: Main Stage
>
>               Matrix     7              3
>               Vector     7              1
>            Index Set     2              2
>    Star Forest Graph     1              0
>        Krylov Solver     1              0
>       Preconditioner     1              0
>
> ========================================================================================================================
> Average time to get PetscTime(): 3.01e-08
> Average time for MPI_Barrier(): 3.8054e-06
> Average time for zero size MPI_Send(): 7.101e-06
> #PETSc Option Table entries:
> -log_view # (source: command line)
> -mat_type mpiaijkokkos # (source: command line)
> -vec_type kokkos # (source: command line)
> #End of PETSc Option Table entries
> Compiled without FORTRAN kernels
> Compiled with full precision matrices (default)
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
> sizeof(PetscScalar) 8 sizeof(PetscInt) 4
> Configure options: COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
> FCOPTFLAGS=-O3 HIPOPTFLAGS=-O3 --with-debugging=0 --with-cc=cc
> --with-cxx=CC --with-fc=ftn --with-hip --with-hipc=hipcc
> --LIBS="-L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib -lmpi
> -L/opt/cray/pe/mpich/8.1.23/gtl/lib -lmpi_gtl_hsa" --download-kokkos
> --download-kokkos-kernels --download-suitesparse --download-hypre
> --download-cmake
> -----------------------------------------
> Libraries compiled on 2024-03-05 17:04:36 on login08
> Machine characteristics:
> Linux-5.14.21-150400.24.46_12.0.83-cray_shasta_c-x86_64-with-glibc2.3.4
> Using PETSc directory: /autofs/nccs-svm1_home1/vanellam/Software/petsc
> Using PETSc arch: arch-linux-frontier-opt-gcc
> -----------------------------------------
>
> Using C compiler: cc  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas
> -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector
> -fvisibility=hidden -O3
> Using Fortran compiler: ftn  -fPIC -Wall -ffree-line-length-none
> -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -O3
> -----------------------------------------
>
> Using include paths:
> -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/include
> -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include
> -I/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/include/suitesparse
> -I/opt/rocm-5.4.0/include
> -----------------------------------------
>
> Using C linker: cc
> Using Fortran linker: ftn
> Using libraries:
> -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
> -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
> -lpetsc
> -Wl,-rpath,/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
> -L/autofs/nccs-svm1_home1/vanellam/Software/petsc/arch-linux-frontier-opt-gcc/lib
> -Wl,-rpath,/opt/rocm-5.4.0/lib -L/opt/rocm-5.4.0/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib
> -L/opt/cray/pe/mpich/8.1.23/ofi/gnu/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.23/gtl/lib
> -L/opt/cray/pe/mpich/8.1.23/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/
> 22.12.1.1/GNU/9.1/x86_64/lib -L/opt/cray/pe/libsci/
> 22.12.1.1/GNU/9.1/x86_64/lib
> -Wl,-rpath,/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib
> -L/sw/frontier/spack-envs/base/opt/cray-sles15-zen3/gcc-12.2.0/darshan-runtime-3.4.0-ftq5gccg3qjtyh5xeo2bz4wqkjayjhw3/lib
> -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib
> -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.1.8/lib
> -L/opt/cray/pe/pmi/6.1.8/lib
> -Wl,-rpath,/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64
> -L/opt/cray/xpmem/2.6.2-2.5_2.22__gd067c3f.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0
> -L/opt/cray/pe/gcc/12.2.0/snos/lib/gcc/x86_64-suse-linux/12.2.0
> -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib64
> -L/opt/cray/pe/gcc/12.2.0/snos/lib64 -Wl,-rpath,/opt/rocm-5.4.0/llvm/lib
> -L/opt/rocm-5.4.0/llvm/lib -Wl,-rpath,/opt/cray/pe/gcc/12.2.0/snos/lib
> -L/opt/cray/pe/gcc/12.2.0/snos/lib -lHYPRE -lspqr -lumfpack -lklu -lcholmod
> -lamd -lkokkoskernels -lkokkoscontainers -lkokkoscore -lkokkossimd
> -lhipsparse -lhipblas -lhipsolver -lrocsparse -lrocsolver -lrocblas
> -lrocrand -lamdhip64 -lmpi -lmpi_gtl_hsa -ldarshan -lz -ldl -lxpmem
> -lgfortran -lm -lmpifort_gnu_91 -lmpi_gnu_91 -lsci_gnu_82_mpi -lsci_gnu_82
> -ldsmml -lpmi -lpmi2 -lgfortran -lquadmath -lpthread -lm -lgcc_s -lstdc++
> -lquadmath -lmpi -lmpi_gtl_hsa
> -----------------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240319/f1e2cbb2/attachment-0001.html>


More information about the petsc-users mailing list