[petsc-users] Performance problem using COO interface

[Fackler, Philip]

In Xolotl's feature-petsc-kokkos branch I have ported the code to use petsc's COO interface for creating the Jacobian matrix (and the Kokkos interface for interacting with Vec entries). As the attached plots show for one case, while the code for computing the RHSFunction and RHSJacobian perform similarly (or slightly better) after the port, the performance for the solve as a whole is significantly worse.

Note:
This is all CPU-only (so kokkos and kokkos-kernels are built with only the serial backend).
The dev version is using MatSetValuesStencil with the default implementations for Mat and Vec.
The port version is using MatSetValuesCOO and is run with -dm_mat_type aijkokkos -dm_vec_type kokkos​.
The port/def version is using MatSetValuesCOO and is run with -dm_vec_type kokkos​ (using the default Mat implementation).

So, this seems to be due be a performance difference in the petsc implementations. Please advise. Is this a known issue? Or am I missing something?

Thank you for the help,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230117/bd9cf904/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NE_4 solve comparison.png
Type: image/png
Size: 16880 bytes
Desc: NE_4 solve comparison.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230117/bd9cf904/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NE_4 rhsfunction and rhsjacobian comparison.png
Type: image/png
Size: 21407 bytes
Desc: NE_4 rhsfunction and rhsjacobian comparison.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230117/bd9cf904/attachment-0003.png>