[petsc-users] hypre on gpus

Fri Nov 19 10:30:40 CST 2021

You should run with -options_left to check that your options are being
used. It may be -mat_type hypre.

I have tested this:

petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre
-ksp_view -pc_type hypre*

You can add -log_view and that will print performance data for each method
like KSPSolve.

If you are configured with a GPU there will be some extra columns that give
the percent of flops on the GPU.

In the past hypre has not registered flops with us, but I do get flops from
hypre now, and -ksp_view showed that it did indeed use hypre.
I saw that the flops were 100% GPU in hypre.

On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalingam at stfc.ac.uk> wrote:

> Hello,
>
>
>
> I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on
> gpus
>
>
>
> ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre
> -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type  boomeramg
> -ksp_monitor -log_view
>
>
>
> I profiled the run using NSYS -  attached you find all the relevant files.
>
> Looking at the profile I doubt if hypre is running on gpus. The cuda
> kernels are barely active.
>
> I don’t see any cuda kernel relevant to solve.
>
> Is my assessment correct? How can I verify if hypre indeed is running on
> gpus?
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Mark Adams <mfadams at lbl.gov>
> *Date: *Friday, 8 October 2021 at 18:47
> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" <
> karthikeyan.chockalingam at stfc.ac.uk>
> *Cc: *"petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] hypre on gpus
>
>
>
> I think you would want to use 'cuda' vec_type, but I .
>
> You might ask Hypre how one verifies that the GPU is used.
>
> Mark
>
>
>
> On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalingam at stfc.ac.uk> wrote:
>
> Yes, I used it for both cpu and gpu. Is that not okay?
>
>
>
> For gpu: -dm_mat_type hypre -dm_vec_type mpicuda
>
>
>
> For cpu:  -dm_mat_type hypre -dm_vec_type mpi
>
>
>
> *From: *Mark Adams <mfadams at lbl.gov>
> *Date: *Friday, 8 October 2021 at 18:28
> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" <
> karthikeyan.chockalingam at stfc.ac.uk>
> *Cc: *"petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] hypre on gpus
>
>
>
> Did you use -dm_mat_type hypre on the GPU case ?
>
>
>
> On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalingam at stfc.ac.uk> wrote:
>
> I tried a different exercise ran the same problem on two cpu cores and on
> two gpu:
>
>
>
> On gpu
>
>
>
> PCApply                6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 6.0e+00 15  0  0  0  1  15  0  0  0  1     0       0      0 0.00e+00    5
> 9.65e+01  0
>
>
>
> and on cpu
>
>
>
> PCApply                6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 16  0  0  0  0  16  0  0  0  0     0
>
>
>
> timings again are close but gpu version did a reduction 6.0e+00 but the
> cpu version did not 0.0e+00.
>
> I am not sure if that is any indication if hypre ran on gpus?
>
>
>
> Thanks,
>
> Karthik.
>
>
>
>
>
> *From: *Mark Adams <mfadams at lbl.gov>
> *Date: *Friday, 8 October 2021 at 16:36
> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" <
> karthikeyan.chockalingam at stfc.ac.uk>
> *Cc: *"petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] hypre on gpus
>
>
>
>
>
>
>
> On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalingam at stfc.ac.uk> wrote:
>
> The PCApply timing on
>
>
>
> gpu
>
>
>
> PCApply                6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 39  0  0  0  0  39  0  0  0  0     0       0      0 0.00e+00    0
> 0.00e+00  0
>
>
>
> and cpu
>
>
>
> PCApply                6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 41  0  0  0  0  41  0  0  0  0     0
>
>
>
>
>
> You don't have GPUs. probably.
>
> Use -dm_mat_type hypre.
>
>
>
> are close. It is hard for me tell if hypre on gpu is on or not.
>
>
>
> Best,
>
> Karthik.
>
>
>
>
>
> *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" <
> karthikeyan.chockalingam at stfc.ac.uk>
> *Date: *Friday, 8 October 2021 at 14:55
> *To: *Mark Adams <mfadams at lbl.gov>
> *Cc: *"petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] hypre on gpus
>
>
>
> Thanks Mark, I will try your recommendations.
>
> Should I also change -dm_vec_type to hypre currently I have it as mpicuda?
>
>
>
> Karthik.
>
>
>
>
>
> *From: *Mark Adams <mfadams at lbl.gov>
> *Date: *Friday, 8 October 2021 at 14:33
> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" <
> karthikeyan.chockalingam at stfc.ac.uk>
> *Cc: *"petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
> *Subject: *Re: [petsc-users] hypre on gpus
>
>
>
> Hypre does not record its flops with PETSc's timers.
>
> Configure with and without CUDA and see if the timings change in PCApply.
>
> Hypre does not dynamically switch between CUDA and CPU solves at
> this time, but you want to use -dm_mat_type hypre.
>
> Mark
>
>
>
> On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalingam at stfc.ac.uk> wrote:
>
> Hello,
>
>
>
> I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have
> attached the python configuration file and -log_view output from running
> the below command options
>
>
>
> mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169
> -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type
> hypre -pc_hypre_type  boomeramg -ksp_gmres_restart 31
> -pc_hypre_boomeramg_strong_threshold 0.7  -ksp_monitor
>
>
>
> The problem was solved and converged but from the output file I suspect
> hypre is not running on gpus as PCApply and DMCreate does *not* record
> any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are
> running on gpus.
>
>
>
> Can you please let me know if I need to add any extra flag to the attached
> arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus?
>
>
>
> Thanks,
>
> Karthik.
>
>
>
>
>
> This email and any attachments are intended solely for the use of the
> named recipients. If you are not the intended recipient you must not use,
> disclose, copy or distribute this email or any of its attachments and
> should notify the sender immediately and delete this email from your
> system. UK Research and Innovation (UKRI) has taken every reasonable
> precaution to minimise risk of this email or any attachments containing
> viruses or malware but the recipient should carry out its own virus and
> malware checks before opening the attachments. UKRI does not accept any
> liability for any losses or damages which the recipient may sustain due to
> presence of any viruses.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211119/d6a369de/attachment.html>