[petsc-users] Using multiple MPI ranks with COO interface crashes in some cases

Mon Nov 14 23:16:50 CST 2022

Hi, Philip,
  Can you tell me instructions to build Xolotl to reproduce the error?
--Junchao Zhang

On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use
> the COO interface for preallocating and setting values in the Jacobian
> matrix. I have found that with some of our test cases, using more than one
> MPI rank results in a crash. Way down in the preconditioner code in petsc a
> Mat gets computed that has "null" for the "productsymbolic" member of its
> "ops". It's pretty far removed from where we compute the Jacobian entries,
> so I haven't been able (so far) to track it back to an error in my code.
> I'd appreciate some help with this from someone who is more familiar with
> the petsc guts so we can figure out what I'm doing wrong. (I'm assuming
> it's a bug in Xolotl.)
>
> Note that this is using the kokkos backend for Mat and Vec in petsc, but
> with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only
> multiple MPI rank run.
>
> Here's a paste of the error output showing the relevant parts of the call
> stack:
>
> [ERROR] [0]PETSC ERROR:
> [ERROR] --------------------- Error Message
> --------------------------------------------------------------
> [ERROR] [1]PETSC ERROR:
> [ERROR] --------------------- Error Message
> --------------------------------------------------------------
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] No support for this operation for this object type
> [ERROR] [1]PETSC ERROR:
> [ERROR] No support for this operation for this object type
> [ERROR] [0]PETSC ERROR:
> [ERROR] No method productsymbolic for Mat of type (null)
> [ERROR] No method productsymbolic for Mat of type (null)
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] See https://petsc.org/release/faq/ for trouble shooting.
> [ERROR] See https://petsc.org/release/faq/ for trouble shooting.
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT
> Date: 2022-10-28 14:39:41 +0000
> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT
> Date: 2022-10-28 14:39:41 +0000
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01 2022
> [ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01 2022
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
> --with-shared-libraries
> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
> --with-shared-libraries
> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #3 MatProductSymbolic() at
> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
> [ERROR] #3 MatProductSymbolic() at
> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #4 MatProduct_Private() at
> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
> [ERROR] #4 MatProduct_Private() at
> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #5 MatMatMult() at
> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
> [ERROR] #5 MatMatMult() at
> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #6 PCGAMGOptProlongator_AGG() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
> [ERROR] #6 PCGAMGOptProlongator_AGG() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #7 PCSetUp_GAMG() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
> [ERROR] #7 PCSetUp_GAMG() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #8 PCSetUp() at
> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
> [ERROR] #8 PCSetUp() at
> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #9 KSPSetUp() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
> [ERROR] #9 KSPSetUp() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #10 KSPSolve_Private() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
> [ERROR] #10 KSPSolve_Private() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #11 KSPSolve() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
> [ERROR] #11 KSPSolve() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #12 PCApply_FieldSplit() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
> [ERROR] #12 PCApply_FieldSplit() at
> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #13 PCApply() at
> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
> [ERROR] #13 PCApply() at
> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #14 KSP_PCApply() at
> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
> [ERROR] #14 KSP_PCApply() at
> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #15 KSPFGMRESCycle() at
> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
> [ERROR] #15 KSPFGMRESCycle() at
> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #16 KSPSolve_FGMRES() at
> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
> [ERROR] #16 KSPSolve_FGMRES() at
> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #17 KSPSolve_Private() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
> [ERROR] #17 KSPSolve_Private() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #18 KSPSolve() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
> [ERROR] #18 KSPSolve() at
> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
> [ERROR] [0]PETSC ERROR:
> [ERROR] [1]PETSC ERROR:
> [ERROR] #19 SNESSolve_NEWTONLS() at
> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
> [ERROR] #19 SNESSolve_NEWTONLS() at
> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #20 SNESSolve() at
> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
> [ERROR] #20 SNESSolve() at
> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #21 TSStep_ARKIMEX() at
> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
> [ERROR] #21 TSStep_ARKIMEX() at
> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
> [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
> [ERROR] [1]PETSC ERROR:
> [ERROR] [0]PETSC ERROR:
> [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
> [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
> [ERROR] PetscSolver::solve: TSSolve failed.
> [ERROR] PetscSolver::solve: TSSolve failed.
> Aborting.
> Aborting.
>
>
>
> Thanks for the help,
>
>
> *Philip Fackler *
> Research Software Engineer, Application Engineering Group
> Advanced Computing Systems Research Section
> Computer Science and Mathematics Division
> *Oak Ridge National Laboratory*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221114/5116b1b9/attachment-0001.html>