[petsc-users] Using PETSc GPU backend

[Matthew Knepley]

On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <cho at slac.stanford.edu> wrote:

> I managed to pass the following options to PETSc using a GPU node on
> Perlmutter.
>
>     -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
>
> o #PETSc Option Table entries:
> -log_view
> -mat_type aijcusparse
>    -options_left
>    -vec_type cuda
>    #End of PETSc Option Table entries
>    WARNING! There are options you set that were not used!
>    WARNING! could be spelling mistake, etc!
>    There is one unused database option. It is:
>    Option left: name:-mat_type value: aijcusparse
>
> The -mat_type option has not been used. In the application code, we use
>
>     ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
>              d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
>
>
If you create the Mat this way, then you need MatSetFromOptions() in order
to set the type from the command line.

  Thanks,

     Matt


> o The percent flops on the GPU for KSPSolve is 17%.
>
> In comparison with a CPU run using 16 MPI tasks, the GPU run is an order
> of magnitude slower. How can I improve the GPU performance?
>
> Thanks,
> Cho
> ------------------------------
> *From:* Ng, Cho-Kuen <cho at slac.stanford.edu>
> *Sent:* Friday, June 30, 2023 7:57 AM
> *To:* Barry Smith <bsmith at petsc.dev>; Mark Adams <mfadams at lbl.gov>
> *Cc:* Matthew Knepley <knepley at gmail.com>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Barry, Mark and Matt,
>
> Thank you all for the suggestions. I will modify the code so we can pass
> runtime options.
>
> Cho
> ------------------------------
> *From:* Barry Smith <bsmith at petsc.dev>
> *Sent:* Friday, June 30, 2023 7:01 AM
> *To:* Mark Adams <mfadams at lbl.gov>
> *Cc:* Matthew Knepley <knepley at gmail.com>; Ng, Cho-Kuen <
> cho at slac.stanford.edu>; petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
>
>   Note that options like -mat_type aijcusparse  -vec_type cuda only work
> if the program is set up to allow runtime swapping of matrix and vector
> types. If you have a call to MatCreateMPIAIJ() or other specific types then
> then these options do nothing but because Mark had you use -options_left
> the program will tell you at the end that it did not use the option so you
> will know.
>
> On Jun 30, 2023, at 9:30 AM, Mark Adams <mfadams at lbl.gov> wrote:
>
> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args
> and you run:
>
> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Mark
>
> On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Mark,
>
> The application code reads in parameters from an input file, where we can
> put the PETSc runtime options. Then we pass the options to
> PetscInitialize(...). Does that sounds right?
>
>
> PETSc will read command line argument automatically in PetscInitialize()
> unless you shut it off.
>
>   Thanks,
>
>     Matt
>
>
> Cho
> ------------------------------
> *From:* Ng, Cho-Kuen <cho at slac.stanford.edu>
> *Sent:* Thursday, June 29, 2023 8:32 PM
> *To:* Mark Adams <mfadams at lbl.gov>
> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Mark,
>
> Thanks for the information. How do I put the runtime options for the
> executable, say, a.out, which does not have the provision to append
> arguments? Do I need to change the C++ main to read in the options?
>
> Cho
> ------------------------------
> *From:* Mark Adams <mfadams at lbl.gov>
> *Sent:* Thursday, June 29, 2023 5:55 PM
> *To:* Ng, Cho-Kuen <cho at slac.stanford.edu>
> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Run with options: -mat_type aijcusparse -vec_type cuda -log_view
> -options_left
>
> The last column of the performance data (from -log_view) will be the
> percent flops on the GPU. Check that that is > 0.
>
> The end of the output will list the options that were used and options
> that were _not_ used (if any). Check that there are no options left.
>
> Mark
>
> On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and
> used it by "spack load petsc/fwge6pf". Then I compiled the application
> code (purely CPU code) linking to the petsc package, hoping that I can get
> performance improvement using the petsc GPU backend. However, the timing
> was the same using the same number of MPI tasks with and without GPU
> accelerators. Have I missed something in the process, for example, setting
> up PETSc options at runtime to use the GPU backend?
>
> Thanks,
> Cho
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230714/23a67afe/attachment.html>