[petsc-users] Kokkos PtAP error

[Mark Adams]

I am getting this error.
This is in GAMG/HEM setup. PtAP for the coarse grid construction works, but
I call this in a graph routine
(/global/u2/m/madams/petsc/src/mat/coarsen/impls/hem/hem.c:1043).

Also, this PtAP does not need to be on the GPU anyway because P is
extremely sparse ... can I pin, say P, to the CPU to keep this all on the
host?

Thanks,
Mark


[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Unspecified symbolic phase for product AB with A
mpiaijkokkos, B mpiaij. Call MatProductSetFromOptions() first
[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the
program crashed before usage or a spelling mistake, etc!
[0]PETSC ERROR:   Option left: name:-ksp_converged_reason (no value)
source: command line
[0]PETSC ERROR:   Option left: name:-ksp_viewxx (no value) source: command
line
[0]PETSC ERROR:   Option left: name:-log_view_gpu_timexxx (no value)
source: command line
[0]PETSC ERROR:   Option left: name:-options_left (no value) source:
command line
[0]PETSC ERROR:   Option left: name:-pc_gamg_use_aggressive_square_graph
value: true source: command line
[0]PETSC ERROR:   Option left: name:-pc_gamg_use_minimum_degree_ordering
value: false source: command line
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.20.0-168-ga7898f52c39
 GIT Date: 2023-10-28 10:07:38 -0500
[0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/./ex13 on a
arch-perlmutter-dbg-gcc-kokkos-cuda named nid001680 by madams Tue Oct 31
06:21:25 2023
[0]PETSC ERROR: Configure options --CFLAGS="   -g" --CXXFLAGS=" -g"
--CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC
--with-fc=ftn --LDFLAGS=-lmpifort_gnu_91 --FFLAGS="   -g " --COPTFLAGS="
-O0" --CXXOPTFLAGS=" -O0" --FOPTFLAGS="   -O0" --download-triangle=1
--with-debugging=1 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec="srun
-G4" --with-batch=0 --download-kokkos --download-kokkos-kernels
--with-kokkos-kernels-tpl=0 --with-make-np=8
PETSC_ARCH=arch-perlmutter-dbg-gcc-kokkos-cuda
[0]PETSC ERROR: #1 MatProductSymbolic() at
/global/u2/m/madams/petsc/src/mat/interface/matproduct.c:807
[0]PETSC ERROR: #2 MatProductSymbolic_PtAP_Unsafe() at
/global/u2/m/madams/petsc/src/mat/interface/matproduct.c:73
[0]PETSC ERROR: #3 MatProductSymbolic_Unsafe() at
/global/u2/m/madams/petsc/src/mat/interface/matproduct.c:185
[0]PETSC ERROR: #4 MatProductSymbolic() at
/global/u2/m/madams/petsc/src/mat/interface/matproduct.c:795
[0]PETSC ERROR: #5 MatPtAP() at
/global/u2/m/madams/petsc/src/mat/interface/matrix.c:9938
[0]PETSC ERROR: #6 MatCoarsenApply_HEM_private() at
/global/u2/m/madams/petsc/src/mat/coarsen/impls/hem/hem.c:1043
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231031/d0ef58ba/attachment.html>