[petsc-users] Proper GPU usage in PETSc

[Mark Adams]

>
>
>
> 12:46 master= ~/petsc/src/snes/tutorials$ make
> PETSC_DIR=/ccs/home/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-cuda-omp -f
> mymake run NP=4 EXTRA='-dm_refine 2 -dm_view'
> jsrun -n 1 -a 4 -c 4 -g 1 -r 1 --smpiargs "-gpu" ./ex13 -dm_plex_box_dim 3
> -dm_plex_box_simplex 0 -potential_petscspace_degree 1 -dm_refine 1
> -ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -pc_type gamg
> -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 1000
> -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
> -mg_levels_ksp_type chebyshev -mg_levels_ksp_max_it 1
> -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type jacobi
> -ksp_monitor -dm_refine 2 -dm_view -vec_type cuda -mat_type aijcusparse
> DM Object: Mesh 4 MPI processes
>   type: plex
> Mesh in 3 dimensions:
>
>
>
> * 0-cells: 125 0 0 0  1-cells: 300 0 0 0  2-cells: 240 0 0 0  3-cells: 64
> 0 0 0*
> Labels:
>   celltype: 4 strata with value/size (0 (125), 1 (300), 4 (240), 7 (64))
>   depth: 4 strata with value/size (0 (125), 1 (300), 2 (240), 3 (64))
>   marker: 1 strata with value/size (1 (378))
>   Face Sets: 6 strata with value/size (6 (49), 5 (49), 3 (49), 4 (49), 1
> (49), 2 (49)
>

This is not getting distributed.

14:24 master= ~/petsc/src/snes/tutorials$ make
PETSC_DIR=/ccs/home/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-cuda-omp -f
mymake run NP=4 EXTRA='-dm_plex_box_faces 4,4,4 -dm_distribute -dm_view'
jsrun -n 1 -a 4 -c 4 -g 1 -r 1 --smpiargs "-gpu" ./ex13 -dm_plex_box_dim 3
-dm_plex_box_simplex 0 -potential_petscspace_degree 1 -dm_refine 1
-ksp_type cg -ksp_rtol 1.e-11 -ksp_norm_type unpreconditioned -pc_type gamg
-pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 1000
-pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0
-mg_levels_ksp_type chebyshev -mg_levels_ksp_max_it 1
-mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10
-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type jacobi
-ksp_monitor *-dm_plex_box_faces 4,4,4 -dm_distribute *-dm_view -vec_type
cuda -mat_type aijcusparse
DM Object: Mesh 4 MPI processes
  type: plex
Mesh in 3 dimensions:



*  0-cells: 225 225 225 225  1-cells: 560 560 560 560  2-cells: 464 464 464
464  3-cells: 128 128 128 128*
Labels:
  celltype: 4 strata with value/size (0 (225), 1 (560), 4 (464), 7 (128))
  depth: 4 strata with value/size (0 (225), 1 (560), 2 (464), 3 (128))
  marker: 1 strata with value/size (1 (384))
  Face Sets: 4 strata with value/size (1 (72), 3 (72), 5 (36), 6 (36))
    0 KSP Residual norm 3.405958130078e+00
    1 KSP Residual norm 3.963373201766e-01
    2 KSP Residual norm 1.282320967427e-01
    3 KSP Residual norm 2.824310691528e-02
    4 KSP Residual norm 7.599375477471e-03
    5 KSP Residual norm 2.083060123161e-03
    6 KSP Residual norm 3.808511520034e-04
    7 KSP Residual norm 8.175282902639e-05
    8 KSP Residual norm 2.175564241206e-05
    9 KSP Residual norm 5.595617772296e-06
   10 KSP Residual norm 1.577629043326e-06
   11 KSP Residual norm 4.401090414293e-07
   12 KSP Residual norm 9.285166456648e-08
   13 KSP Residual norm 1.606071620113e-08
   14 KSP Residual norm 3.371039707655e-09
   15 KSP Residual norm 7.260259461122e-10
   16 KSP Residual norm 1.511146527931e-10
   17 KSP Residual norm 3.804907486855e-11
   18 KSP Residual norm 1.062507039176e-11
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200924/3d5b6d2b/attachment.html>