[petsc-users] Unexpected performance losses switching to COO interface

[Junchao Zhang]

Hi, Philip,
  I will look into the tarballs and get back to you.
   Thanks.
--Junchao Zhang


On Mon, Oct 2, 2023 at 9:41 AM Fackler, Philip via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> We finally have xolotl ported to use the new COO interface and the
> aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port
> to our previous version (using MatSetValuesStencil and the default Mat and
> Vec implementations), we expected to see an improvement in performance for
> both the "serial" and "cuda" builds (here I'm referring to the kokkos
> configuration).
>
> Attached are two plots that show timings for three different cases. All of
> these were run on Ascent (the Summit-like training system) with 6 MPI tasks
> (on a single node). The CUDA cases were given one GPU per task (and used
> CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases
> we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent
> as possible.
>
> The performance of RHSJacobian (where the bulk of computation happens in
> xolotl) behaved basically as expected (better than expected in the serial
> build). NE_3 case in CUDA was the only one that performed worse, but not
> surprisingly, since its workload for the GPUs is much smaller. We've still
> got more optimization to do on this.
>
> The real surprise was how much worse the overall solve times were. This
> seems to be due simply to switching to the kokkos-based implementation. I'm
> wondering if there are any changes we can make in configuration or runtime
> arguments to help with PETSc's performance here. Any help looking into this
> would be appreciated.
>
> The tarballs linked here
> <https://drive.google.com/file/d/19X_L3SVkGBM9YUzXnRR_kVWFG0JFwqZ3/view?usp=drive_link>
> and here
> <https://drive.google.com/file/d/15yDBN7-YlO1g6RJNPYNImzr611i1Ffhv/view?usp=drive_link>
> are profiling databases which, once extracted, can be viewed with
> hpcviewer. I don't know how helpful that will be, but hopefully it can give
> you some direction.
>
> Thanks for your help,
>
>
> *Philip Fackler *
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> *Oak Ridge National Laboratory*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231002/48f455d1/attachment.html>