[petsc-users] Performance problem using COO interface

[Zhang, Junchao]

Hi, Philip,
  Could you add -log_view and see what functions are used in the solve? Since it is CPU-only, perhaps with -log_view of different runs, we can easily see which functions slowed down.

--Junchao Zhang
________________________________
From: Fackler, Philip <facklerpw at ornl.gov>
Sent: Tuesday, January 17, 2023 4:13 PM
To: xolotl-psi-development at lists.sourceforge.net <xolotl-psi-development at lists.sourceforge.net>; petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Cc: Mills, Richard Tran <rtmills at anl.gov>; Zhang, Junchao <jczhang at mcs.anl.gov>; Blondel, Sophie <sblondel at utk.edu>; Roth, Philip <rothpc at ornl.gov>
Subject: Performance problem using COO interface

In Xolotl's feature-petsc-kokkos branch I have ported the code to use petsc's COO interface for creating the Jacobian matrix (and the Kokkos interface for interacting with Vec entries). As the attached plots show for one case, while the code for computing the RHSFunction and RHSJacobian perform similarly (or slightly better) after the port, the performance for the solve as a whole is significantly worse.

Note:
This is all CPU-only (so kokkos and kokkos-kernels are built with only the serial backend).
The dev version is using MatSetValuesStencil with the default implementations for Mat and Vec.
The port version is using MatSetValuesCOO and is run with -dm_mat_type aijkokkos -dm_vec_type kokkos​.
The port/def version is using MatSetValuesCOO and is run with -dm_vec_type kokkos​ (using the default Mat implementation).

So, this seems to be due be a performance difference in the petsc implementations. Please advise. Is this a known issue? Or am I missing something?

Thank you for the help,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230117/5369ab27/attachment.html>