<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in
the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code.
I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Here's a paste of the error output showing the relevant parts of the call stack:</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
[ERROR] [0]PETSC ERROR:
<div class="ContentPasted0">[ERROR] --------------------- Error Message --------------------------------------------------------------</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] --------------------- Error Message --------------------------------------------------------------</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] No support for this operation for this object type</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] No support for this operation for this object type</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] No method productsymbolic for Mat of type (null)</div>
<div class="ContentPasted0">[ERROR] No method productsymbolic for Mat of type (null)</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] See https://petsc.org/release/faq/ for trouble shooting.</div>
<div class="ContentPasted0">[ERROR] See https://petsc.org/release/faq/ for trouble shooting.</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000</div>
<div class="ContentPasted0">[ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022</div>
<div class="ContentPasted0">[ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
--with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install</div>
<div class="ContentPasted0">[ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
--with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918</div>
<div class="ContentPasted0">[ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138</div>
<div class="ContentPasted0">[ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793</div>
<div class="ContentPasted0">[ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820</div>
<div class="ContentPasted0">[ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897</div>
<div class="ContentPasted0">[ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769</div>
<div class="ContentPasted0">[ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639</div>
<div class="ContentPasted0">[ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994</div>
<div class="ContentPasted0">[ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406</div>
<div class="ContentPasted0">[ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825</div>
<div class="ContentPasted0">[ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div class="ContentPasted0">[ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246</div>
<div class="ContentPasted0">[ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441</div>
<div class="ContentPasted0">[ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380</div>
<div class="ContentPasted0">[ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152</div>
<div class="ContentPasted0">[ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273</div>
<div class="ContentPasted0">[ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899</div>
<div class="ContentPasted0">[ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div class="ContentPasted0">[ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210</div>
<div class="ContentPasted0">[ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689</div>
<div class="ContentPasted0">[ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791</div>
<div class="ContentPasted0">[ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445</div>
<div class="ContentPasted0">[ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445</div>
<div class="ContentPasted0">[ERROR] [1]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] [0]PETSC ERROR: </div>
<div class="ContentPasted0">[ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836</div>
<div class="ContentPasted0">[ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836</div>
<div class="ContentPasted0">[ERROR] PetscSolver::solve: TSSolve failed.</div>
<div class="ContentPasted0">[ERROR] PetscSolver::solve: TSSolve failed.</div>
<div class="ContentPasted0">Aborting.</div>
Aborting.<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Thanks for the help,<br>
</div>
<div class="elementToProof">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>