[petsc-dev] PETSc amg solver with gpu seems run slowly

[Barry Smith]

  It is using 

MatSOR               369 1.0 9.1214e+00 1.0 7.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 29 27  0  0  0  29 27  0  0  0   803       0      0 0.00e+00  565 1.35e+03  0

which runs on the CPU not the GPU hence the large amount of time in memory copies and poor performance. We are switching the default to be Chebyshev/Jacobi which runs completely on the GPU (may already be switched in the main branch). 

You can run with -mg_levels_pc_type jacobi You should then see almost the entire solver running on the GPU.

You may need to tune the number of smoothing steps or other parameters of GAMG to get the faster solution time.

  Barry


> On Mar 22, 2022, at 10:30 AM, Qi Yang <qiyang at oakland.edu> wrote:
> 
> To whom it may concern,
> 
> I have tried petsc ex50(Possion) with cuda, ksp cg solver and gamg precondition, however, it run for about 30s. I also tried NVIDIA AMGX with the same solver and same grid (3000*3000), it only took 2s. I used nsight system software to analyze those two cases, found petsc took much time in the memory process (63% of total time, however, amgx only took 19%). Attached are screenshots of them.
> 
> The petsc command is : mpiexec -n 1 ./ex50  -da_grid_x 3000 -da_grid_y 3000 -ksp_type cg -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -vec_type cuda -mat_type aijcusparse -ksp_monitor -ksp_view -log-view
> 
> The log file is also attached.
> 
> Regards,
> Qi
> 
> <1.png>
> <2.png>
> <log.PETSc_cg_amg_ex50_gpu_cuda>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220322/25c084d4/attachment.html>