[petsc-users] hypre on gpus

Mon Nov 22 07:50:41 CST 2021

Hi Matt,

Below is the entire error message:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------

[0]PETSC ERROR: Object is in wrong state

[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange()

[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021

[0]PETSC ERROR: ./ex4 on a  named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021

[0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90

[0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784

[0]PETSC ERROR: #2 main() at ex4.c:40

[0]PETSC ERROR: PETSc Option Table entries:

[0]PETSC ERROR: -ksp_type cg

[0]PETSC ERROR: -ksp_view

[0]PETSC ERROR: -mat_type hypre

[0]PETSC ERROR: -pc_type hypre

[0]PETSC ERROR: -use_gpu_aware_mpi 0

[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------

--------------------------------------------------------------------------

I have also attached the make.log. Thank you for having a look.

Best,
Karthik.

From: Matthew Knepley <knepley at gmail.com>
Date: Monday, 22 November 2021 at 13:41
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk>
Cc: Mark Adams <mfadams at lbl.gov>, "petsc-users at mcs.anl.gov" <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] hypre on gpus

On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:

Thank you for your response. I tried to run the same example

petsc/src/ksp/ksp/tutorials$  ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre

but it crashed with the below error

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange()

Hi Karthik,

Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error,
since that should be the same on any architecture. So could you also send the make.log?

  Thanks,

    Matt

Below are the options used to configure hypre with cuda support. Do you spot any mistakes?

--with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl

--with-cuda=1

--with-cuda-arch=70

--download-hypre=yes

--download-hypre-configure-arguments=HYPRE_CUDA_SM=70

--download-hypre-commit=origin/hypre_petsc

--with-shared-libraries=1

--known-mpi-shared-libraries=1

--with-cc=mpicc

--with-cxx=mpicxx

-with-fc=mpif90

Best,

Karthik.

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Date: Friday, 19 November 2021 at 16:31
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

You should run with -options_left to check that your options are being used. It may be -mat_type hypre.

I have tested this:

petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre

You can add -log_view and that will print performance data for each method like KSPSolve.

If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU.

In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre.
I saw that the flops were 100% GPU in hypre.

On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:
Hello,

I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus

./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type  boomeramg  -ksp_monitor -log_view

I profiled the run using NSYS -  attached you find all the relevant files.
Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active.
I don’t see any cuda kernel relevant to solve.
Is my assessment correct? How can I verify if hypre indeed is running on gpus?

Best,
Karthik.

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Date: Friday, 8 October 2021 at 18:47
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

I think you would want to use 'cuda' vec_type, but I .
You might ask Hypre how one verifies that the GPU is used.
Mark

On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:
Yes, I used it for both cpu and gpu. Is that not okay?

For gpu: -dm_mat_type hypre -dm_vec_type mpicuda

For cpu:  -dm_mat_type hypre -dm_vec_type mpi

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Date: Friday, 8 October 2021 at 18:28
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

Did you use -dm_mat_type hypre on the GPU case ?

On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:
I tried a different exercise ran the same problem on two cpu cores and on two gpu:

On gpu

PCApply                6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15  0  0  0  1  15  0  0  0  1     0       0      0 0.00e+00    5 9.65e+01  0

and on cpu

PCApply                6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16  0  0  0  0  16  0  0  0  0     0

timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00.
I am not sure if that is any indication if hypre ran on gpus?

Thanks,
Karthik.

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Date: Friday, 8 October 2021 at 16:36
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:
The PCApply timing on

gpu

PCApply                6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39  0  0  0  0  39  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0

and cpu

PCApply                6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41  0  0  0  0  41  0  0  0  0     0

You don't have GPUs. probably.
Use -dm_mat_type hypre.

are close. It is hard for me tell if hypre on gpu is on or not.

Best,
Karthik.

From: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Date: Friday, 8 October 2021 at 14:55
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

Thanks Mark, I will try your recommendations.
Should I also change -dm_vec_type to hypre currently I have it as mpicuda?

Karthik.

From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Date: Friday, 8 October 2021 at 14:33
To: "Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>>
Cc: "petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>" <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] hypre on gpus

Hypre does not record its flops with PETSc's timers.
Configure with and without CUDA and see if the timings change in PCApply.
Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre.
Mark

On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI <karthikeyan.chockalingam at stfc.ac.uk<mailto:karthikeyan.chockalingam at stfc.ac.uk>> wrote:
Hello,

I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options

mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169  -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type  boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7  -ksp_monitor

The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus.

Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus?

Thanks,
Karthik.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211122/ff5f9df9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 118056 bytes
Desc: make.log
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211122/ff5f9df9/attachment-0001.obj>