[petsc-users] Proper GPU usage in PETSc

Zhang, Chonglin zhangc20 at rpi.edu
Thu Sep 24 12:25:49 CDT 2020



On Sep 24, 2020, at 1:11 PM, Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>> wrote:



On Sep 24, 2020, at 11:48 AM, Zhang, Chonglin <zhangc20 at rpi.edu<mailto:zhangc20 at rpi.edu>> wrote:

Thanks Mark and Barry!

A quick try of using “-pc_type jacobi” did reduce the number of count for “CpuToGpu” and “GpuToCpu”, although using “-pc_type gamg” (the counts did not decrease in this case) solves the problem faster (may not be of any meaning since the problem size is too small; the function “DMPlexCreateFromCellListParallelPetsc()" is slow for large problem size preventing running larger problems, separate issue).

Would this “CpuToGpu” and “GpuToCpu” data transfer contribute a significant amount of time for a realistic sized problem, say for example a linear problem with ~1-2 million DOFs?

   It depends on how often the copies are done.

   With GAMG once the preconditioner is built the entire linear solve can run on the GPU and Mark has some good speed ups of the liner solve using GAMG on the GPU instead of the CPU on Summit.

   The speedup of the entire simulation will depend on the relative cost of the finite element matrix assembly vs the linear solver time and Amdahl's law kicks in so, for example, if the finite element assembly takes 50 percent of the time even if the linear solve takes 0 time one cannot only get a speedup of two which is not much.


Thanks for the detailed explanation Barry!

Mark: could you share the results of GAMG on GPU vs CPU on Summit, or pointing to me where I could see them. (Actual code how you are doing this would be even better as a learning opportunity for me). Thanks!


Also, is there any plan to have the SNES and DMPlex code run on GPU?

  Basically the finite element computation for the nonlinear function and its Jacobian need to run on the GPU, this is a big project that we've barely begun thinking about. If this is something you are interested in it would be fantastic if you could take a look at that.

I see. I will think about this, discuss internally and get back to you if I can!

Thanks!
Chonglin


  Barry




Thanks!
Chonglin

On Sep 24, 2020, at 12:17 PM, Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>> wrote:


   MatSOR() runs on the CPU, this causes copy to CPU for each application of MatSOR() and then a copy to GPU for the next step.

   You can try, for example -pc_type jacobi  better yet use PCGAMG if it amenable for your problem.

   Also the problem is way to small for a GPU.

  There will be copies between the GPU/CPU for each SNES iteration since the DMPLEX code does not run on GPUs.

   Barry



On Sep 24, 2020, at 10:08 AM, Zhang, Chonglin <zhangc20 at rpi.edu<mailto:zhangc20 at rpi.edu>> wrote:

Dear PETSc Users,

I have some questions regarding the proper GPU usage. I would like to know the proper way to:
(1) solve linear equation in SNES, using GPU in PETSc; what syntax/arguments should I be using;
(2) how to avoid/reduce the “CpuToGpu count” and “GpuToCpu count” data transfer showed in PETSc log file, when using CUDA aware MPI.


Details of what I am doing now and my observations are below:

System and compilers used:
(1) RPI’s AiMOS computer (node wise, it is the same as Summit);
(2) using GCC 7.4.0 and Spectrum-MPI 10.3.

I am doing the followings to solve the linear Poisson equation with SNES interface, under DMPlex:
(1) using DMPlex to set up the unstructured mesh;
(2) using DM to create vector and matrix;
(3) using SNES interface to solve the linear Poisson equation, with “-snes_type ksponly”;
(4) using “dm_vec_type cuda”, “dm_mat_type aijcusparse “ to use GPU vector and matrix, as suggested in this webpage: https://www.mcs.anl.gov/petsc/features/gpus.html
(5) using “use_gpu_aware_mpi” with PETSc, and using `mpirun -gpu` to enable GPU-Direct ( similar as "srun --smpiargs=“-gpu”" for Summit): https://secure.cci.rpi.edu/wiki/Slurm/#gpu-direct; https://www.olcf.ornl.gov/wp-content/uploads/2018/11/multi-gpu-workshop.pdf
(6) using “-options_left” to check and make sure all the arguments are accepted and used by PETSc.
(7) After problem setup, I am running the “SNESSolve()” multiple times to solve the linear problem and observe the log file with “-log_view"

I noticed that if I run “SNESSolve()” 500 times, instead of 50 times, the “CpuToGpu count” and/or “GpuToCpu count” increased roughly 10 times for some of the operations: SNESSolve, MatSOR, VecMDot, VecCUDACopyTo, VecCUDACopyFrom, MatCUSPARSCopyTo. See below for a truncated log corresponding to running SNESSolve() 500 times:


Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   - GpuToCpu - GPU
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size   Count   Size  %F
---------------------------------------------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided        510 1.0 4.9205e-03 1.1 0.00e+00 0.0 3.5e+01 4.0e+00 1.0e+03  0  0  0  0  0   0  0 21  0  0     0       0      0 0.00e+00    0 0.00e+00  0
BuildTwoSidedF       501 1.0 1.0199e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+03  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0
SNESSolve            500 1.0 3.2570e+02 1.0 1.18e+10 1.0 0.0e+00 0.0e+00 8.7e+05100100  0  0100 100100  0  0100   144     202   31947 7.82e+02 63363 1.44e+03 82
SNESSetUp              1 1.0 6.0082e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0
SNESFunctionEval     500 1.0 3.9826e+01 1.0 3.60e+08 1.0 0.0e+00 0.0e+00 5.0e+02 12  3  0  0  0  12  3  0  0  0    36      13      0 0.00e+00 1000 2.48e+01  0
SNESJacobianEval     500 1.0 4.8200e+01 1.0 5.97e+08 1.0 0.0e+00 0.0e+00 2.0e+03 15  5  0  0  0  15  5  0  0  0    50       0   1000 7.77e+01  500 1.24e+01  0
DMPlexResidualFE     500 1.0 3.6923e+01 1.1 3.56e+08 1.0 0.0e+00 0.0e+00 0.0e+00 10  3  0  0  0  10  3  0  0  0    39       0      0 0.00e+00  500 1.24e+01  0
DMPlexJacobianFE     500 1.0 4.6013e+01 1.0 5.95e+08 1.0 0.0e+00 0.0e+00 2.0e+03 14  5  0  0  0  14  5  0  0  0    52       0   1000 7.77e+01    0 0.00e+00  0
MatSOR             30947 1.0 3.1254e+00 1.1 1.21e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1 10  0  0  0   1 10  0  0  0  1542       0      0 0.00e+00 61863 1.41e+03  0
MatAssemblyBegin     511 1.0 5.3428e+00256.4 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+03  1  0  0  0  0   1  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0
MatAssemblyEnd       511 1.0 4.3440e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.1e+01  0  0  0  0  0   0  0  0  0  0     0       0   1002 7.80e+01    0 0.00e+00  0
MatCUSPARSCopyTo    1002 1.0 3.6557e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0   1002 7.80e+01    0 0.00e+00  0
VecMDot            29930 1.0 3.7843e+01 1.0 2.62e+09 1.0 0.0e+00 0.0e+00 6.0e+04 12 22  0  0  7  12 22  0  0  7   277    3236   29930 6.81e+02    0 0.00e+00 100
VecNorm            31447 1.0 2.1164e+01 1.4 1.79e+08 1.0 0.0e+00 0.0e+00 6.3e+04  5  2  0  0  7   5  2  0  0  7    34      55   1017 2.31e+01    0 0.00e+00 100
VecNormalize       30947 1.0 2.3957e+01 1.1 2.65e+08 1.0 0.0e+00 0.0e+00 6.2e+04  7  2  0  0  7   7  2  0  0  7    44      51   1017 2.31e+01    0 0.00e+00 100
VecCUDACopyTo      30947 1.0 7.8866e+00 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  2  0  0  0  0   2  0  0  0  0     0       0   30947 7.04e+02    0 0.00e+00  0
VecCUDACopyFrom    63363 1.0 1.0873e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00 63363 1.44e+03  0
KSPSetUp             500 1.0 2.2737e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 5.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0
KSPSolve             500 1.0 2.3687e+02 1.0 1.08e+10 1.0 0.0e+00 0.0e+00 8.6e+05 72 92  0  0 99  73 92  0  0 99   182     202   30947 7.04e+02 61863 1.41e+03 89
KSPGMRESOrthog     29930 1.0 1.8920e+02 1.0 7.87e+09 1.0 0.0e+00 0.0e+00 6.4e+05 58 67  0  0 74  58 67  0  0 74   166     209   29930 6.81e+02    0 0.00e+00 100
PCApply            30947 1.0 3.1555e+00 1.1 1.21e+09 1.0 0.0e+00 0.0e+00 0.0e+00  1 10  0  0  0   1 10  0  0  0  1527       0      0 0.00e+00 61863 1.41e+03  0


Thanks!
Chonglin




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200924/4b265fa1/attachment-0001.html>


More information about the petsc-users mailing list