[petsc-users] [EXTERNAL] Re: Initializing kokkos before petsc causes a problem

[Fackler, Philip]

Thanks Junchao! Your fix worked perfectly.

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
________________________________
From: Fackler, Philip via Xolotl-psi-development <xolotl-psi-development at lists.sourceforge.net>
Sent: Tuesday, June 27, 2023 11:46
To: Junchao Zhang <junchao.zhang at gmail.com>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>; xolotl-psi-development at lists.sourceforge.net <xolotl-psi-development at lists.sourceforge.net>
Subject: Re: [Xolotl-psi-development] [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem

OK, great! I'll try it out soon.

Thank you,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
________________________________
From: Junchao Zhang <junchao.zhang at gmail.com>
Sent: Tuesday, June 27, 2023 10:58
To: Fackler, Philip <facklerpw at ornl.gov>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>; Blondel, Sophie <sblondel at utk.edu>; xolotl-psi-development at lists.sourceforge.net <xolotl-psi-development at lists.sourceforge.net>
Subject: Re: [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem

Hi, Philip,
   It's my fault.  I should follow up early that this problem was fixed by https://gitlab.com/petsc/petsc/-/merge_requests/6586<https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_6586&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=lI7woCCMo6nlYg0cHYuCi4LIvG19JU7v2iPr4uuN0is_2tl07L6cmUlXIvdqNz5_&s=ZW5VibonWkK9G9LcvP3951WBPqqjC81SKj26aPC3u48&e=>.
   Could you try petsc/main?

   Thanks.
--Junchao Zhang


On Tue, Jun 27, 2023 at 9:30 AM Fackler, Philip <facklerpw at ornl.gov<mailto:facklerpw at ornl.gov>> wrote:
Good morning Junchao! I'm following up here to see if there is any update to petsc to resolve this issue, or if we need to come up with a work-around.

Thank you,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
________________________________
From: Junchao Zhang <junchao.zhang at gmail.com<mailto:junchao.zhang at gmail.com>>
Sent: Wednesday, June 7, 2023 22:45
To: Fackler, Philip <facklerpw at ornl.gov<mailto:facklerpw at ornl.gov>>
Cc: petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov> <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>>; Blondel, Sophie <sblondel at utk.edu<mailto:sblondel at utk.edu>>; xolotl-psi-development at lists.sourceforge.net<mailto:xolotl-psi-development at lists.sourceforge.net> <xolotl-psi-development at lists.sourceforge.net<mailto:xolotl-psi-development at lists.sourceforge.net>>
Subject: [EXTERNAL] Re: [petsc-users] Initializing kokkos before petsc causes a problem

Hi, Philip,
  Thanks for reporting. I will have a look at the issue.
--Junchao Zhang


On Wed, Jun 7, 2023 at 9:30 AM Fackler, Philip via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
I'm encountering a problem in xolotl. We initialize kokkos before initializing petsc. Therefore...

The pointer referenced here:
https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx#L363<https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_blob_main_src_vec_is_sf_impls_basic_kokkos_sfkok.kokkos.cxx-23L363&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=mm5WnvW66V9aFezISyisTVEya_MgbI4YjpAuorp4BHdklRfAD8kTzaDBY3VlzI-4&s=i_BmubxwJ4005UQUtS2UbTyKmZwbDAq7aIfA-T4Gnsg&e=>
<https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_blob_main_include_petsc-5Fkokkos.hpp&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=mm5WnvW66V9aFezISyisTVEya_MgbI4YjpAuorp4BHdklRfAD8kTzaDBY3VlzI-4&s=DWQqfHRdrciXObsAIEgF4mGXbMH1xx2vIpcM07ncznU&e=>

from here:
https://gitlab.com/petsc/petsc/-/blob/main/include/petsc_kokkos.hpp<https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_blob_main_include_petsc-5Fkokkos.hpp&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=mm5WnvW66V9aFezISyisTVEya_MgbI4YjpAuorp4BHdklRfAD8kTzaDBY3VlzI-4&s=DWQqfHRdrciXObsAIEgF4mGXbMH1xx2vIpcM07ncznU&e=>

remains null because the code to initialize it is skipped here:
https://gitlab.com/petsc/petsc/-/blob/main/src/sys/objects/kokkos/kinit.kokkos.cxx#L28<https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_blob_main_src_sys_objects_kokkos_kinit.kokkos.cxx-23L28&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=mm5WnvW66V9aFezISyisTVEya_MgbI4YjpAuorp4BHdklRfAD8kTzaDBY3VlzI-4&s=qjM7d6j2BTI04hPQnjucl2H1osvGvortk9kn9e20Hww&e=>
See line 71.

Can this be modified to allow for kokkos to have been initialized by the application before initializing petsc?

Thank you for your help,

Philip Fackler
Research Software Engineer, Application Engineering Group
Advanced Computing Systems Research Section
Computer Science and Mathematics Division
Oak Ridge National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230721/28ad19b8/attachment-0001.html>