[petsc-users] [EXTERNAL] Re: Kokkos backend for Mat and Vec diverging when running on CUDA device.

[Junchao Zhang]

Can you paste -log_view result so I can see what functions are used?

--Junchao Zhang


On Tue, Nov 15, 2022 at 10:24 AM Fackler, Philip <facklerpw at ornl.gov> wrote:

> Yes, most (but not all) of our system test cases fail with the kokkos/cuda
> or cuda backends. All of them pass with the CPU-only kokkos backend.
>
>
> *Philip Fackler *
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> *Oak Ridge National Laboratory*
> ------------------------------
> *From:* Junchao Zhang <junchao.zhang at gmail.com>
> *Sent:* Monday, November 14, 2022 19:34
> *To:* Fackler, Philip <facklerpw at ornl.gov>
> *Cc:* xolotl-psi-development at lists.sourceforge.net <
> xolotl-psi-development at lists.sourceforge.net>; petsc-users at mcs.anl.gov <
> petsc-users at mcs.anl.gov>; Blondel, Sophie <sblondel at utk.edu>; Zhang,
> Junchao <jczhang at mcs.anl.gov>; Roth, Philip <rothpc at ornl.gov>
> *Subject:* [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec
> diverging when running on CUDA device.
>
> Hi, Philip,
>   Sorry to hear that.  It seems you could run the same code on CPUs but
> not no GPUs (with either petsc/Kokkos backend or petsc/cuda backend, is it
> right?
>
> --Junchao Zhang
>
>
> On Mon, Nov 14, 2022 at 12:13 PM Fackler, Philip via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> This is an issue I've brought up before (and discussed in-person with
> Richard). I wanted to bring it up again because I'm hitting the limits of
> what I know to do, and I need help figuring this out.
>
> The problem can be reproduced using Xolotl's "develop" branch built
> against a petsc build with kokkos and kokkos-kernels enabled. Then, either
> add the relevant kokkos options to the "petscArgs=" line in the system test
> parameter file(s), or just replace the system test parameter files with the
> ones from the "feature-petsc-kokkos" branch. See here the files that
> begin with "params_system_".
>
> Note that those files use the "kokkos" options, but the problem is similar
> using the corresponding cuda/cusparse options. I've already tried building
> kokkos-kernels with no TPLs and got slightly different results, but the
> same problem.
>
> Any help would be appreciated.
>
> Thanks,
>
>
> *Philip Fackler *
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> *Oak Ridge National Laboratory*
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221115/42a84143/attachment.html>