[petsc-users] Using PETSc GPU backend
Ng, Cho-Kuen
cho at slac.stanford.edu
Fri Aug 11 23:53:07 CDT 2023
Barry,
I tried again today on Perlmutter and running on multiple GPU nodes worked. Likely, I had messed up something the other day. Also, I was able to have multiple MPI tasks on a GPU using Nvidia MPS. The petsc output shows the number of MPI tasks:
KSP Object: 32 MPI processes
Can petsc show the number of GPUs used?
Thanks,
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev>
Sent: Wednesday, August 9, 2023 4:09 PM
To: Ng, Cho-Kuen <cho at slac.stanford.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Using PETSc GPU backend
We would need more information about "hanging". Do PETSc examples and tiny problems "hang" on multiple nodes? If you run with -info what are the last messages printed? Can you run with a debugger to see where it is "hanging"?
On Aug 9, 2023, at 5:59 PM, Ng, Cho-Kuen <cho at slac.stanford.edu> wrote:
Barry and Matt,
Thanks for your help. Now I can use petsc GPU backend on Perlmutter: 1 node, 4 MPI tasks and 4 GPUs. However, I ran into problems with multiple nodes: 2 nodes, 8 MPI tasks and 8 GPUs. The run hung on KSPSolve. How can I fix this?
Best,
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, July 17, 2023 6:58 AM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
The examples that use DM, in particular DMDA all trivially support using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda
On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
Barry,
Thank you so much for the clarification.
I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are there other tutorials available?
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Saturday, July 15, 2023 8:36 AM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Cho,
We currently have a crappy API for turning on GPU support, and our documentation is misleading in places.
People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect.
This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda
If you use DM to create the matrices and vectors then you can use -dm_mat_type aijcusparse -dm_vec_type cuda
Sorry for the confusion.
Barry
On Jul 15, 2023, at 8:03 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
Matt,
After inserting 2 lines in the code:
ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
ierr = MatSetFromOptions(A);CHKERRQ(ierr);
ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
"There are no unused options." However, there is no improvement on the GPU performance.
1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual.
2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions().
THanks,
Matt
Thanks,
Cho
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, July 14, 2023 5:57 PM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>; Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
I managed to pass the following options to PETSc using a GPU node on Perlmutter.
-mat_type aijcusparse -vec_type cuda -log_view -options_left
Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
o #PETSc Option Table entries:
-log_view
-mat_type aijcusparse
-options_left
-vec_type cuda
#End of PETSc Option Table entries
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
There is one unused database option. It is:
Option left: name:-mat_type value: aijcusparse
The -mat_type option has not been used. In the application code, we use
ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line.
Thanks,
Matt
o The percent flops on the GPU for KSPSolve is 17%.
In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance?
Thanks,
Cho
________________________________
From: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Sent: Friday, June 30, 2023 7:57 AM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>; Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Barry, Mark and Matt,
Thank you all for the suggestions. I will modify the code so we can pass runtime options.
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, June 30, 2023 7:01 AM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
On Jun 30, 2023, at 9:30 AM, Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:
a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left
Mark
On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Mark,
The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?
PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.
Thanks,
Matt
Cho
________________________________
From: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Sent: Thursday, June 29, 2023 8:32 PM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Mark,
Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options?
Cho
________________________________
From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: Thursday, June 29, 2023 5:55 PM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.
The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.
Mark
On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend?
Thanks,
Cho
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Monday, July 17, 2023 6:58 AM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
The examples that use DM, in particular DMDA all trivially support using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda
On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
Barry,
Thank you so much for the clarification.
I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are there other tutorials available?
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Saturday, July 15, 2023 8:36 AM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Cho,
We currently have a crappy API for turning on GPU support, and our documentation is misleading in places.
People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect.
This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda
If you use DM to create the matrices and vectors then you can use -dm_mat_type aijcusparse -dm_vec_type cuda
Sorry for the confusion.
Barry
On Jul 15, 2023, at 8:03 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
Matt,
After inserting 2 lines in the code:
ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
ierr = MatSetFromOptions(A);CHKERRQ(ierr);
ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
"There are no unused options." However, there is no improvement on the GPU performance.
1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual.
2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions().
THanks,
Matt
Thanks,
Cho
________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Friday, July 14, 2023 5:57 PM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>; Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>> wrote:
I managed to pass the following options to PETSc using a GPU node on Perlmutter.
-mat_type aijcusparse -vec_type cuda -log_view -options_left
Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
o #PETSc Option Table entries:
-log_view
-mat_type aijcusparse
-options_left
-vec_type cuda
#End of PETSc Option Table entries
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
There is one unused database option. It is:
Option left: name:-mat_type value: aijcusparse
The -mat_type option has not been used. In the application code, we use
ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line.
Thanks,
Matt
o The percent flops on the GPU for KSPSolve is 17%.
In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance?
Thanks,
Cho
________________________________
From: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Sent: Friday, June 30, 2023 7:57 AM
To: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>; Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Barry, Mark and Matt,
Thank you all for the suggestions. I will modify the code so we can pass runtime options.
Cho
________________________________
From: Barry Smith <bsmith at petsc.dev<mailto:bsmith at petsc.dev>>
Sent: Friday, June 30, 2023 7:01 AM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>; Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>; petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
On Jun 30, 2023, at 9:30 AM, Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> wrote:
PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:
a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left
Mark
On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Mark,
The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?
PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.
Thanks,
Matt
Cho
________________________________
From: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Sent: Thursday, June 29, 2023 8:32 PM
To: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Mark,
Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options?
Cho
________________________________
From: Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>>
Sent: Thursday, June 29, 2023 5:55 PM
To: Ng, Cho-Kuen <cho at slac.stanford.edu<mailto:cho at slac.stanford.edu>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>
Subject: Re: [petsc-users] Using PETSc GPU backend
Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left
The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.
The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.
Mark
On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend?
Thanks,
Cho
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230812/c6443430/attachment-0001.html>
More information about the petsc-users
mailing list