[petsc-users] Using Sundials from PETSc

[Vanella, Marcos (Fed)]

Hi Matt, very interesting project you are working on. We haven't gone deep on how we would do this in GPUs and are starting to look at options. We will explore if it is possible to batch work needed for several cells within a thread group on the gpu.

We use a single Cartesian mesh per MPI process (usually with 40^3 to 50^3 cells). Something I implemented to avoid the MPI process over-subscription of GPU with PETSc solvers was to cluster several MPI Processes per GPU on resource sets. Then, the processes in the set would pass matrix (at setup) and RHS to a single process (set master) which communicates with the GPU.
The GPU solution is then brought back to the set master which distributes it to the MPI processes in the set as needed.
So, only a set of processes as large as the number of GPUs in the calculation (with their own MPI communicator) call the PETSc matrix and vector building, and solve routines.  The neat thing is that all MPI communications are local to the node. This idea is not new, it was developed by the researchers at GWU that interfaced PETSc to AMGx back when there were no native GPU solvers in PETSc, HYPRE and other libs (~2016).

Best,
Marcos

________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Monday, October 16, 2023 4:31 PM
To: Vanella, Marcos (Fed) <marcos.vanella at nist.gov>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>; Paul, Chandan (IntlAssoc) <chandan.paul at nist.gov>
Subject: Re: [petsc-users] Using Sundials from PETSc

On Mon, Oct 16, 2023 at 4:08 PM Vanella, Marcos (Fed) <marcos.vanella at nist.gov<mailto:marcos.vanella at nist.gov>> wrote:
Hi Mathew, we have code that time splits the combustion step from the chemical species transport, so on each computational cell for each fluid flow time step, once transport is done we have the mixture chemical composition as initial condition. We are looking into doing finite rate chemistry with skeletal combustion models (20+ equations) in each cell for each fluid time step. Sundials provides the CVODE solver for the time integration of these, and would be interesting to see if we can make use of GPU acceleration. From their User Guide for Version 6.6.0 there are several GPU implementations for building RHS and using linear, nonlinear and stiff ODE solvers.

We are doing a similar thing in CHREST (https://www.buffalo.edu/chrest.html). Since we normally use hundreds of species and thousands of reactions for the reduced mechanism, we are using TChem2 to build and solve the system in each cell.

Since these systems are so small, you are likely to need some way of batching them within a warp. Do you have an idea for this already?

  Thanks,

     Matt

Thank you Satish for the comment. Might be better at this point to first get an idea on what the implementation in our code using Sundials directly would look like. Then, we can see if it is possible and makes sense to access it through PETSc.
We have things working in CPU making use of and older version of CVODE.

BTW after some changes in our code we are starting running larger cases using GPU accelerated iterative solvers from PETSc, so we have PETSc interfaced already.

Thanks!

________________________________
From: Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>>
Sent: Monday, October 16, 2023 3:03 PM
To: Vanella, Marcos (Fed) <marcos.vanella at nist.gov<mailto:marcos.vanella at nist.gov>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>; Paul, Chandan (IntlAssoc) <chandan.paul at nist.gov<mailto:chandan.paul at nist.gov>>
Subject: Re: [petsc-users] Using Sundials from PETSc

On Mon, Oct 16, 2023 at 2:29 PM Vanella, Marcos (Fed) via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hi, we were wondering if it would be possible to call the latest version of Sundials from PETSc?

The short answer is, no. We are at v2.5 and they are at v6.5. There were no dates on the version history page, so I do not know how out of date we are. There have not been any requests for update until now.

We would be happy to get an MR for the updates if you want to try it.

We are interested in doing chemistry using GPUs and already have interfaces to PETSc from our code.

How does the GPU interest interact with the SUNDIALS version?

  Thanks,

     Matt

Thanks,
Marcos


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231016/fac9afdb/attachment-0001.html>