[petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases
Mark Adams
mfadams at lbl.gov
Wed Nov 16 07:05:39 CST 2022
I can not build right now on Crusher or Perlmutter but I saw this on both.
Here is an example output using src/snes/tests/ex13.c using the appended
.petscrc
This uses 64 processors and the 8 processor case worked. This has been
semi-nondertminisitc for me.
(and I have attached my current Perlmutter problem)
Hope this helps,
Mark
-dm_plex_simplex 0
-dm_plex_dim 3
-dm_plex_box_lower 0,0,0
-dm_plex_box_upper 1,1,1
-petscpartitioner_simple_process_grid 2,2,2
-potential_petscspace_degree 2
-snes_max_it 1
-ksp_max_it 200
-ksp_type cg
-ksp_rtol 1.e-12
-ksp_norm_type unpreconditioned
-snes_rtol 1.e-8
#-pc_type gamg
#-pc_gamg_type agg
#-pc_gamg_agg_nsmooths 1
-pc_gamg_coarse_eq_limit 100
-pc_gamg_process_eq_limit 400
-pc_gamg_reuse_interpolation true
#-snes_monitor
#-ksp_monitor_short
-ksp_converged_reason
#-ksp_view
#-snes_converged_reason
#-mg_levels_ksp_max_it 2
-mg_levels_ksp_type chebyshev
#-mg_levels_ksp_type richardson
#-mg_levels_ksp_richardson_scale 0.8
-mg_levels_pc_type jacobi
-pc_gamg_esteig_ksp_type cg
-pc_gamg_esteig_ksp_max_it 10
-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
-dm_distribute
-petscpartitioner_type simple
-pc_gamg_repartition false
-pc_gamg_coarse_grid_layout_type compact
-pc_gamg_threshold 0.01
#-pc_gamg_threshold_scale .5
-pc_gamg_aggressive_coarsening 1
#-check_pointer_intensity 0
-snes_type ksponly
#-mg_coarse_sub_pc_factor_mat_solver_type cusparse
#-info :pc
#-use_gpu_aware_mpi 1
-options_left
#-malloc_debug
-benchmark_it 10
#-pc_gamg_use_parallel_coarse_grid_solver
#-mg_coarse_pc_type jacobi
#-mg_coarse_ksp_type cg
#-mg_coarse_ksp_rtol 1.e-2
#-mat_cusparse_transgen
-snes_lag_jacobian -2
On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <junchao.zhang at gmail.com>
wrote:
> Mark,
> Do you have a reproducer using petsc examples?
>
> On Tue, Nov 15, 2022, 12:49 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>> Junchao, this is the same problem that I have been having right?
>>
>> On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> I built petsc with:
>>>
>>> $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug
>>> --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0
>>> --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices
>>> --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos
>>> --download-kokkos-kernels
>>>
>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
>>>
>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
>>>
>>>
>>> Then I build xolotl in a separate build directory (after checking out
>>> the "feature-petsc-kokkos" branch) with:
>>>
>>> $ cmake -DCMAKE_BUILD_TYPE=Debug
>>> -DKokkos_DIR=$HOME/build/petsc/debug/install
>>> -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
>>>
>>> $ make -j4 SystemTester
>>>
>>>
>>> Then, from the xolotl build directory, run (for example):
>>>
>>> $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
>>>
>>> Note that this test case will use the parameter file
>>> '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line
>>> arguments for petsc in its "petscArgs=..." line. If you look at
>>> '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases
>>> follow the same naming convention with their corresponding parameter files
>>> under '<xolotl-src>/benchmarks'.
>>>
>>> The failure happens with the NE_4 case (which is 2D) and the PSI_3 case
>>> (which is 1D).
>>>
>>> Let me know if this is still unclear.
>>>
>>> Thanks,
>>>
>>>
>>> *Philip Fackler *
>>> Research Software Engineer, Application Engineering Group
>>> Advanced Computing Systems Research Section
>>> Computer Science and Mathematics Division
>>> *Oak Ridge National Laboratory*
>>> ------------------------------
>>> *From:* Junchao Zhang <junchao.zhang at gmail.com>
>>> *Sent:* Tuesday, November 15, 2022 00:16
>>> *To:* Fackler, Philip <facklerpw at ornl.gov>
>>> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>; Blondel,
>>> Sophie <sblondel at utk.edu>
>>> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with
>>> COO interface crashes in some cases
>>>
>>> Hi, Philip,
>>> Can you tell me instructions to build Xolotl to reproduce the error?
>>> --Junchao Zhang
>>>
>>>
>>> On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <
>>> petsc-users at mcs.anl.gov> wrote:
>>>
>>> In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use
>>> the COO interface for preallocating and setting values in the Jacobian
>>> matrix. I have found that with some of our test cases, using more than one
>>> MPI rank results in a crash. Way down in the preconditioner code in petsc a
>>> Mat gets computed that has "null" for the "productsymbolic" member of its
>>> "ops". It's pretty far removed from where we compute the Jacobian entries,
>>> so I haven't been able (so far) to track it back to an error in my code.
>>> I'd appreciate some help with this from someone who is more familiar with
>>> the petsc guts so we can figure out what I'm doing wrong. (I'm assuming
>>> it's a bug in Xolotl.)
>>>
>>> Note that this is using the kokkos backend for Mat and Vec in petsc, but
>>> with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only
>>> multiple MPI rank run.
>>>
>>> Here's a paste of the error output showing the relevant parts of the
>>> call stack:
>>>
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] --------------------- Error Message
>>> --------------------------------------------------------------
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] --------------------- Error Message
>>> --------------------------------------------------------------
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] No support for this operation for this object type
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] No support for this operation for this object type
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] No method productsymbolic for Mat of type (null)
>>> [ERROR] No method productsymbolic for Mat of type (null)
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting.
>>> [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting.
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT
>>> Date: 2022-10-28 14:39:41 +0000
>>> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT
>>> Date: 2022-10-28 14:39:41 +0000
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01
>>> 2022
>>> [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01
>>> 2022
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
>>> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
>>> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
>>> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
>>> --with-shared-libraries
>>> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
>>> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
>>> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
>>> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
>>> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
>>> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
>>> --with-shared-libraries
>>> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
>>> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
>>> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
>>> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #3 MatProductSymbolic() at
>>> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
>>> [ERROR] #3 MatProductSymbolic() at
>>> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #4 MatProduct_Private() at
>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
>>> [ERROR] #4 MatProduct_Private() at
>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #5 MatMatMult() at
>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
>>> [ERROR] #5 MatMatMult() at
>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #6 PCGAMGOptProlongator_AGG() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
>>> [ERROR] #6 PCGAMGOptProlongator_AGG() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #7 PCSetUp_GAMG() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
>>> [ERROR] #7 PCSetUp_GAMG() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #8 PCSetUp() at
>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
>>> [ERROR] #8 PCSetUp() at
>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #9 KSPSetUp() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
>>> [ERROR] #9 KSPSetUp() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #10 KSPSolve_Private() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
>>> [ERROR] #10 KSPSolve_Private() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #11 KSPSolve() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>> [ERROR] #11 KSPSolve() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #12 PCApply_FieldSplit() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
>>> [ERROR] #12 PCApply_FieldSplit() at
>>> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #13 PCApply() at
>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
>>> [ERROR] #13 PCApply() at
>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #14 KSP_PCApply() at
>>> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
>>> [ERROR] #14 KSP_PCApply() at
>>> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #15 KSPFGMRESCycle() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
>>> [ERROR] #15 KSPFGMRESCycle() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #16 KSPSolve_FGMRES() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
>>> [ERROR] #16 KSPSolve_FGMRES() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #17 KSPSolve_Private() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
>>> [ERROR] #17 KSPSolve_Private() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #18 KSPSolve() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>> [ERROR] #18 KSPSolve() at
>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] #19 SNESSolve_NEWTONLS() at
>>> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
>>> [ERROR] #19 SNESSolve_NEWTONLS() at
>>> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #20 SNESSolve() at
>>> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
>>> [ERROR] #20 SNESSolve() at
>>> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #21 TSStep_ARKIMEX() at
>>> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
>>> [ERROR] #21 TSStep_ARKIMEX() at
>>> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
>>> [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
>>> [ERROR] [1]PETSC ERROR:
>>> [ERROR] [0]PETSC ERROR:
>>> [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
>>> [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
>>> [ERROR] PetscSolver::solve: TSSolve failed.
>>> [ERROR] PetscSolver::solve: TSSolve failed.
>>> Aborting.
>>> Aborting.
>>>
>>>
>>>
>>> Thanks for the help,
>>>
>>>
>>> *Philip Fackler *
>>> Research Software Engineer, Application Engineering Group
>>> Advanced Computing Systems Research Section
>>> Computer Science and Mathematics Division
>>> *Oak Ridge National Laboratory*
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221116/72ffed1f/attachment-0001.html>
-------------- next part --------------
+ '[' -z '' ']'
+ case "$-" in
+ __lmod_vx=x
+ '[' -n x ']'
+ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted
+ unset __lmod_vx
+ export SLURM_CPU_BIND=cores
+ SLURM_CPU_BIND=cores
+ EXTRA='-dm_view -log_view -use_gpu_aware_mpi 0'
+ KOKKOS_ARGS='-dm_mat_type aijkokkos -mat_type aijkokkos -dm_vec_type kokkos'
+ CUDA_ARGS='-dm_mat_type aijcusparse -mat_type aijcusparse -dm_vec_type cuda'
+ HYPRE_ARGS='-dm_mat_type hypre -dm_vec_type kokkos -pc_type hypre'
+ AMGX_ARGS='-pc_type amgx -pc_amgx_verbose true -pc_amgx_jacobi_relaxation_factor .7 -pc_amgx_aggressive_levels 2 -pc_amgx_print_grid_stats'
+ GAMG_ARGS='-pc_type gamg'
+ SRUN_ARGS='--cpu-bind=cores --ntasks-per-core=1 --gpu-bind=single:2'
+ NR=4
+ N=1
+ NG=4
+ NC=2
+ PG=2
+ date
Sun 30 Oct 2022 05:29:58 AM PDT
+ for REFINE in 3
+ for NPIDX in 2
+ let 'N1 = 2'
+ let 'N2 = 2 * 2'
+ let 'N4 = 4 * 2'
+ let 'REF1 = 3 + 1'
+ let 'NODES = 2 * 2 * 2'
+ let 'N = 8 * 2 * 4'
+ let 'NGPU = 8 * 4'
+ echo n= 64 ' NODES=' 8
n= 64 NODES= 8
++ printf %03d 8
+ foo=008
+ srun -n64 -N8 -G 32 --cpu-bind=cores --ntasks-per-core=1 --gpu-bind=single:2 ../ex13 -dm_plex_box_faces 4,4,4 -petscpartitioner_simple_process_grid 2,2,2 -petscpartitioner_simple_node_grid 2,2,2 -dm_refine 4 -pc_type gamg -dm_mat_type aijkokkos -mat_type aijkokkos -dm_vec_type kokkos -dm_view -log_view -use_gpu_aware_mpi 0
++ tee stdout.log
++ tee stderr.log
DM Object: box 64 MPI processes
type: plex
box in 3 dimensions:
Min/Max of 0-cells per rank: 4913/4913
Min/Max of 1-cells per rank: 13872/13872
Min/Max of 2-cells per rank: 13056/13056
Min/Max of 3-cells per rank: 4096/4096
Labels:
celltype: 4 strata with value/size (0 (4913), 1 (13872), 4 (13056), 7 (4096))
depth: 4 strata with value/size (0 (4913), 1 (13872), 2 (13056), 3 (4096))
marker: 1 strata with value/size (1 (3169))
Face Sets: 3 strata with value/size (1 (961), 3 (961), 6 (961))
Number equations N = 2048383
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: No method productsymbolic for Mat of type (null)
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[0]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[0]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[0]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[0]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[22]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: No support for this operation for this object type
[22]PETSC ERROR: No method productsymbolic for Mat of type (null)
[22]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[22]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[22]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[22]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[22]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[22]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[22]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[38]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[38]PETSC ERROR: No support for this operation for this object type
[38]PETSC ERROR: No method productsymbolic for Mat of type (null)
[38]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[38]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[38]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[38]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[38]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[38]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[38]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[52]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[52]PETSC ERROR: No support for this operation for this object type
[52]PETSC ERROR: No method productsymbolic for Mat of type (null)
[52]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[52]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[52]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[52]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[52]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[52]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[52]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[62]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[62]PETSC ERROR: No support for this operation for this object type
[62]PETSC ERROR: No method productsymbolic for Mat of type (null)
[62]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[62]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[62]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[62]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[62]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[62]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[62]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: No support for this operation for this object type
[8]PETSC ERROR: No method productsymbolic for Mat of type (null)
[8]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[8]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[8]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[8]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[8]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[8]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[0]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[0]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[0]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[0]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[0]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[0]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[0]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[0]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[0]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[22]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[22]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[22]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[38]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[38]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[8]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[8]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[52]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[62]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[62]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[62]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[0]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[0]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[25]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[25]PETSC ERROR: No support for this operation for this object type
[25]PETSC ERROR: No method productsymbolic for Mat of type (null)
[25]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[25]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[25]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[25]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[25]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[25]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[25]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[41]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[41]PETSC ERROR: No support for this operation for this object type
[41]PETSC ERROR: No method productsymbolic for Mat of type (null)
[41]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[41]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[41]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[41]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[41]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[41]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[41]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[0]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[0]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[0]PETSC ERROR: #14 main() at ex13.c:178
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -benchmark_it 10
[0]PETSC ERROR: -dm_distribute
[0]PETSC ERROR: -dm_mat_type aijkokkos
[0]PETSC ERROR: -dm_plex_box_faces 4,4,4
[0]PETSC ERROR: -dm_plex_box_lower 0,0,0
[0]PETSC ERROR: -dm_plex_box_upper 2,2,2
[0]PETSC ERROR: -dm_plex_dim 3
[0]PETSC ERROR: -dm_plex_simplex 0
[0]PETSC ERROR: -dm_refine 4
[0]PETSC ERROR: -dm_vec_type kokkos
[0]PETSC ERROR: -dm_view
[0]PETSC ERROR: -ksp_converged_reason
[0]PETSC ERROR: -ksp_max_it 200
[0]PETSC ERROR: -ksp_norm_type unpreconditioned
[0]PETSC ERROR: -ksp_rtol 1.e-12
[0]PETSC ERROR: -ksp_type cg
[0]PETSC ERROR: -log_view
[0]PETSC ERROR: -mat_type aijkokkos
[0]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[0]PETSC ERROR: -mg_levels_ksp_type chebyshev
[22]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[22]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[22]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[22]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[22]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[22]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[22]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[22]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[22]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[54]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[54]PETSC ERROR: No support for this operation for this object type
[54]PETSC ERROR: No method productsymbolic for Mat of type (null)
[54]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[54]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[54]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[54]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[54]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[54]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[54]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[0]PETSC ERROR: -mg_levels_pc_type jacobi
[0]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[0]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[0]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[0]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[0]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[0]PETSC ERROR: -pc_gamg_process_eq_limit 400
[0]PETSC ERROR: -pc_gamg_repartition false
[0]PETSC ERROR: -pc_gamg_reuse_interpolation true
[0]PETSC ERROR: -pc_gamg_threshold 0.01
[0]PETSC ERROR: -pc_type gamg
[0]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[54]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[54]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[54]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: No support for this operation for this object type
[1]PETSC ERROR: No method productsymbolic for Mat of type (null)
[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[1]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[1]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[1]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[1]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[1]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: No support for this operation for this object type
[2]PETSC ERROR: No method productsymbolic for Mat of type (null)
[2]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[2]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[2]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[2]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[2]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[2]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[2]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[2]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[41]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[8]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[8]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[8]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[8]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[8]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[8]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[8]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[8]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[8]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[2]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[38]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[38]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[38]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[38]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[38]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[38]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[38]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[38]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[38]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[43]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[43]PETSC ERROR: No support for this operation for this object type
[43]PETSC ERROR: No method productsymbolic for Mat of type (null)
[43]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[43]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[43]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[43]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[43]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[43]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[43]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[62]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[62]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[62]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[62]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[62]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[62]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[62]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[62]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[62]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: No support for this operation for this object type
[4]PETSC ERROR: No method productsymbolic for Mat of type (null)
[4]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[4]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[4]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[4]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[4]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[4]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[4]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[38]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[38]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[38]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[38]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[38]PETSC ERROR: #14 main() at ex13.c:178
[62]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[62]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[62]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[62]PETSC ERROR: #14 main() at ex13.c:178
[62]PETSC ERROR: PETSc Option Table entries:
[62]PETSC ERROR: -benchmark_it 10
[4]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[4]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[4]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[4]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[4]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[4]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[4]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[4]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[4]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[33]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[33]PETSC ERROR: No support for this operation for this object type
[33]PETSC ERROR: No method productsymbolic for Mat of type (null)
[33]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[33]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[33]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[33]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[8]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[8]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[8]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: No support for this operation for this object type
[6]PETSC ERROR: No method productsymbolic for Mat of type (null)
[6]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[6]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[6]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[6]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[6]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[6]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[6]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: No support for this operation for this object type
[9]PETSC ERROR: No method productsymbolic for Mat of type (null)
[9]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[9]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[9]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[9]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[9]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[9]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[6]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[6]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[6]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[43]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[43]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[43]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[54]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[54]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[54]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[54]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[54]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[54]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[54]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[44]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[44]PETSC ERROR: No support for this operation for this object type
[44]PETSC ERROR: No method productsymbolic for Mat of type (null)
[44]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[44]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[44]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[44]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[44]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[44]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[44]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[8]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[8]PETSC ERROR: #14 main() at ex13.c:178
[8]PETSC ERROR: PETSc Option Table entries:
[8]PETSC ERROR: -benchmark_it 10
[8]PETSC ERROR: -dm_distribute
[8]PETSC ERROR: -dm_mat_type aijkokkos
[8]PETSC ERROR: -dm_plex_box_faces 4,4,4
[8]PETSC ERROR: -dm_plex_box_lower 0,0,0
[8]PETSC ERROR: -dm_plex_box_upper 2,2,2
[8]PETSC ERROR: -dm_plex_dim 3
[8]PETSC ERROR: -dm_plex_simplex 0
[8]PETSC ERROR: -dm_refine 4
[8]PETSC ERROR: -dm_vec_type kokkos
[8]PETSC ERROR: -dm_view
[8]PETSC ERROR: -ksp_converged_reason
[8]PETSC ERROR: -ksp_max_it 200
[8]PETSC ERROR: -ksp_norm_type unpreconditioned
[8]PETSC ERROR: -ksp_rtol 1.e-12
[8]PETSC ERROR: -ksp_type cg
[8]PETSC ERROR: -log_view
[8]PETSC ERROR: -mat_type aijkokkos
[8]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[8]PETSC ERROR: -mg_levels_ksp_type chebyshev
[8]PETSC ERROR: -mg_levels_pc_type jacobi
[8]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[8]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[8]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[8]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[8]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[8]PETSC ERROR: -pc_gamg_process_eq_limit 400
[6]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[6]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[6]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[6]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[6]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[6]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[6]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[6]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[6]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[6]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[6]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[6]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[6]PETSC ERROR: #14 main() at ex13.c:178
[6]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[0]PETSC ERROR: -petscpartitioner_type simple
[0]PETSC ERROR: -potential_petscspace_degree 2
[0]PETSC ERROR: -snes_lag_jacobian -2
[0]PETSC ERROR: -snes_max_it 1
[0]PETSC ERROR: -snes_rtol 1.e-8
[0]PETSC ERROR: -snes_type ksponly
[0]PETSC ERROR: -use_gpu_aware_mpi 0
[0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 0] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[1]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[1]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[1]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[1]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[1]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[1]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[1]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[1]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[1]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[1]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[1]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[1]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[1]PETSC ERROR: #14 main() at ex13.c:178
[1]PETSC ERROR: PETSc Option Table entries:
[1]PETSC ERROR: -benchmark_it 10
[1]PETSC ERROR: -dm_distribute
[1]PETSC ERROR: -dm_mat_type aijkokkos
[1]PETSC ERROR: -dm_plex_box_faces 4,4,4
[1]PETSC ERROR: -dm_plex_box_lower 0,0,0
[1]PETSC ERROR: -dm_plex_box_upper 2,2,2
[1]PETSC ERROR: -dm_plex_dim 3
[1]PETSC ERROR: -dm_plex_simplex 0
[1]PETSC ERROR: -dm_refine 4
[1]PETSC ERROR: -dm_vec_type kokkos
[1]PETSC ERROR: -dm_view
[1]PETSC ERROR: -ksp_converged_reason
[1]PETSC ERROR: -ksp_max_it 200
[1]PETSC ERROR: -ksp_norm_type unpreconditioned
[1]PETSC ERROR: -ksp_rtol 1.e-12
[1]PETSC ERROR: -ksp_type cg
[1]PETSC ERROR: -log_view
[1]PETSC ERROR: -mat_type aijkokkos
[1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[1]PETSC ERROR: -mg_levels_ksp_type chebyshev
[1]PETSC ERROR: -mg_levels_pc_type jacobi
[1]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[1]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[1]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[1]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[1]PETSC ERROR: -pc_gamg_process_eq_limit 400
[1]PETSC ERROR: -pc_gamg_repartition false
[1]PETSC ERROR: -pc_gamg_reuse_interpolation true
[1]PETSC ERROR: -pc_gamg_threshold 0.01
[1]PETSC ERROR: -pc_type gamg
[1]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[1]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[1]PETSC ERROR: -petscpartitioner_type simple
[1]PETSC ERROR: -potential_petscspace_degree 2
[1]PETSC ERROR: -snes_lag_jacobian -2
[1]PETSC ERROR: -snes_max_it 1
[1]PETSC ERROR: -snes_rtol 1.e-8
[1]PETSC ERROR: -snes_type ksponly
[1]PETSC ERROR: -use_gpu_aware_mpi 0
[1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[2]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[2]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[2]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[2]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[2]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[2]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[2]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[2]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[2]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[2]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[22]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[22]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[22]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[22]PETSC ERROR: #14 main() at ex13.c:178
[22]PETSC ERROR: PETSc Option Table entries:
[22]PETSC ERROR: -benchmark_it 10
[22]PETSC ERROR: -dm_distribute
[22]PETSC ERROR: -dm_mat_type aijkokkos
[22]PETSC ERROR: -dm_plex_box_faces 4,4,4
[22]PETSC ERROR: -dm_plex_box_lower 0,0,0
[25]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[52]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[52]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[52]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[52]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[2]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[2]PETSC ERROR: #14 main() at ex13.c:178
[2]PETSC ERROR: PETSc Option Table entries:
[2]PETSC ERROR: -benchmark_it 10
[2]PETSC ERROR: -dm_distribute
[2]PETSC ERROR: -dm_mat_type aijkokkos
[2]PETSC ERROR: -dm_plex_box_faces 4,4,4
[2]PETSC ERROR: -dm_plex_box_lower 0,0,0
[2]PETSC ERROR: -dm_plex_box_upper 2,2,2
[2]PETSC ERROR: -dm_plex_dim 3
[2]PETSC ERROR: -dm_plex_simplex 0
[2]PETSC ERROR: -dm_refine 4
[2]PETSC ERROR: -dm_vec_type kokkos
[2]PETSC ERROR: -dm_view
[2]PETSC ERROR: -ksp_converged_reason
[2]PETSC ERROR: -ksp_max_it 200
[2]PETSC ERROR: -ksp_norm_type unpreconditioned
[2]PETSC ERROR: -ksp_rtol 1.e-12
[2]PETSC ERROR: -ksp_type cg
[2]PETSC ERROR: -log_view
[2]PETSC ERROR: -mat_type aijkokkos
[2]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[2]PETSC ERROR: -mg_levels_ksp_type chebyshev
[2]PETSC ERROR: -mg_levels_pc_type jacobi
[2]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[26]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: No support for this operation for this object type
[26]PETSC ERROR: No method productsymbolic for Mat of type (null)
[26]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[26]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[26]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[26]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[26]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[26]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[26]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[52]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[52]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[52]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[52]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[52]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[52]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[52]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[52]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[52]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[52]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[44]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[44]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[44]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[2]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[2]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[2]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[2]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[2]PETSC ERROR: -pc_gamg_process_eq_limit 400
[2]PETSC ERROR: -pc_gamg_repartition false
[2]PETSC ERROR: -pc_gamg_reuse_interpolation true
[2]PETSC ERROR: -pc_gamg_threshold 0.01
[2]PETSC ERROR: -pc_type gamg
[2]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[2]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[2]PETSC ERROR: -petscpartitioner_type simple
[2]PETSC ERROR: -potential_petscspace_degree 2
[2]PETSC ERROR: -snes_lag_jacobian -2
[2]PETSC ERROR: -snes_max_it 1
[2]PETSC ERROR: -snes_rtol 1.e-8
[2]PETSC ERROR: -snes_type ksponly
[2]PETSC ERROR: -use_gpu_aware_mpi 0
[2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[52]PETSC ERROR: #14 main() at ex13.c:178
[52]PETSC ERROR: PETSc Option Table entries:
[52]PETSC ERROR: -benchmark_it 10
[52]PETSC ERROR: -dm_distribute
[52]PETSC ERROR: -dm_mat_type aijkokkos
[4]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[4]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[4]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[4]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[4]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[4]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[4]PETSC ERROR: #14 main() at ex13.c:178
[4]PETSC ERROR: PETSc Option Table entries:
[4]PETSC ERROR: -benchmark_it 10
[4]PETSC ERROR: -dm_distribute
[4]PETSC ERROR: -dm_mat_type aijkokkos
[4]PETSC ERROR: -dm_plex_box_faces 4,4,4
[4]PETSC ERROR: -dm_plex_box_lower 0,0,0
[4]PETSC ERROR: -dm_plex_box_upper 2,2,2
[4]PETSC ERROR: -dm_plex_dim 3
[4]PETSC ERROR: -dm_plex_simplex 0
[4]PETSC ERROR: -dm_refine 4
[54]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[54]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[54]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[54]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[54]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[54]PETSC ERROR: #14 main() at ex13.c:178
[54]PETSC ERROR: PETSc Option Table entries:
[54]PETSC ERROR: -benchmark_it 10
[54]PETSC ERROR: -dm_distribute
[54]PETSC ERROR: -dm_mat_type aijkokkos
[54]PETSC ERROR: -dm_plex_box_faces 4,4,4
[54]PETSC ERROR: -dm_plex_box_lower 0,0,0
[54]PETSC ERROR: -dm_plex_box_upper 2,2,2
[54]PETSC ERROR: -dm_plex_dim 3
[54]PETSC ERROR: -dm_plex_simplex 0
[54]PETSC ERROR: -dm_refine 4
[54]PETSC ERROR: -dm_vec_type kokkos
[54]PETSC ERROR: -dm_view
[54]PETSC ERROR: -ksp_converged_reason
[4]PETSC ERROR: -dm_vec_type kokkos
[4]PETSC ERROR: -dm_view
[4]PETSC ERROR: -ksp_converged_reason
[4]PETSC ERROR: -ksp_max_it 200
[4]PETSC ERROR: -ksp_norm_type unpreconditioned
[4]PETSC ERROR: -ksp_rtol 1.e-12
[4]PETSC ERROR: -ksp_type cg
[4]PETSC ERROR: -log_view
[4]PETSC ERROR: -mat_type aijkokkos
[4]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[4]PETSC ERROR: -mg_levels_ksp_type chebyshev
[4]PETSC ERROR: -mg_levels_pc_type jacobi
[4]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[4]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[4]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[4]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[4]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[4]PETSC ERROR: -pc_gamg_process_eq_limit 400
[4]PETSC ERROR: -pc_gamg_repartition false
[4]PETSC ERROR: -pc_gamg_reuse_interpolation true
[4]PETSC ERROR: -pc_gamg_threshold 0.01
[4]PETSC ERROR: -pc_type gamg
[4]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[4]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[54]PETSC ERROR: -ksp_max_it 200
[54]PETSC ERROR: -ksp_norm_type unpreconditioned
[54]PETSC ERROR: -ksp_rtol 1.e-12
[54]PETSC ERROR: -ksp_type cg
[4]PETSC ERROR: -petscpartitioner_type simple
[4]PETSC ERROR: -potential_petscspace_degree 2
[4]PETSC ERROR: -snes_lag_jacobian -2
[4]PETSC ERROR: -snes_max_it 1
[4]PETSC ERROR: -snes_rtol 1.e-8
[4]PETSC ERROR: -snes_type ksponly
[4]PETSC ERROR: -use_gpu_aware_mpi 0
[4]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[6]PETSC ERROR: -benchmark_it 10
[6]PETSC ERROR: -dm_distribute
[6]PETSC ERROR: -dm_mat_type aijkokkos
[6]PETSC ERROR: -dm_plex_box_faces 4,4,4
[6]PETSC ERROR: -dm_plex_box_lower 0,0,0
[6]PETSC ERROR: -dm_plex_box_upper 2,2,2
[6]PETSC ERROR: -dm_plex_dim 3
[6]PETSC ERROR: -dm_plex_simplex 0
[6]PETSC ERROR: -dm_refine 4
[6]PETSC ERROR: -dm_vec_type kokkos
[6]PETSC ERROR: -dm_view
[6]PETSC ERROR: -ksp_converged_reason
[6]PETSC ERROR: -ksp_max_it 200
[6]PETSC ERROR: -ksp_norm_type unpreconditioned
[6]PETSC ERROR: -ksp_rtol 1.e-12
[6]PETSC ERROR: -ksp_type cg
[6]PETSC ERROR: -log_view
[6]PETSC ERROR: -mat_type aijkokkos
[6]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[6]PETSC ERROR: -mg_levels_ksp_type chebyshev
[6]PETSC ERROR: -mg_levels_pc_type jacobi
[6]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[6]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[6]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[6]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[6]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[6]PETSC ERROR: -pc_gamg_process_eq_limit 400
[6]PETSC ERROR: -pc_gamg_repartition false
[6]PETSC ERROR: -pc_gamg_reuse_interpolation true
[6]PETSC ERROR: -pc_gamg_threshold 0.01
[6]PETSC ERROR: -pc_type gamg
[6]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[6]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[6]PETSC ERROR: -petscpartitioner_type simple
[6]PETSC ERROR: -potential_petscspace_degree 2
[6]PETSC ERROR: -snes_lag_jacobian -2
[6]PETSC ERROR: -snes_max_it 1
[6]PETSC ERROR: -snes_rtol 1.e-8
[6]PETSC ERROR: -snes_type ksponly
[6]PETSC ERROR: -use_gpu_aware_mpi 0
[6]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 6] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
MPICH ERROR [Rank 1] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
MPICH ERROR [Rank 2] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: No support for this operation for this object type
[3]PETSC ERROR: No method productsymbolic for Mat of type (null)
[3]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[3]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[3]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[3]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[3]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[3]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[3]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[3]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[3]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[3]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[3]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[3]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[3]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[3]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[3]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[3]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[3]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[3]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[3]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[3]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[3]PETSC ERROR: #14 main() at ex13.c:178
[3]PETSC ERROR: PETSc Option Table entries:
[3]PETSC ERROR: -benchmark_it 10
[3]PETSC ERROR: -dm_distribute
[3]PETSC ERROR: -dm_mat_type aijkokkos
[3]PETSC ERROR: -dm_plex_box_faces 4,4,4
[3]PETSC ERROR: -dm_plex_box_lower 0,0,0
[3]PETSC ERROR: -dm_plex_box_upper 2,2,2
[3]PETSC ERROR: -dm_plex_dim 3
[3]PETSC ERROR: -dm_plex_simplex 0
[3]PETSC ERROR: -dm_refine 4
[3]PETSC ERROR: -dm_vec_type kokkos
[3]PETSC ERROR: -dm_view
[3]PETSC ERROR: -ksp_converged_reason
[57]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[57]PETSC ERROR: No support for this operation for this object type
[57]PETSC ERROR: No method productsymbolic for Mat of type (null)
[57]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[57]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[57]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[57]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[57]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[57]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[57]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[9]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[9]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[9]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[9]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
MPICH ERROR [Rank 4] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[9]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[9]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[9]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[9]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[9]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[9]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[9]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[9]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[9]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[9]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: No support for this operation for this object type
[5]PETSC ERROR: No method productsymbolic for Mat of type (null)
[5]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[5]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[5]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[5]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[5]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[5]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[5]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[9]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[9]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[5]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[5]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[5]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[23]PETSC ERROR: No support for this operation for this object type
[23]PETSC ERROR: No method productsymbolic for Mat of type (null)
[23]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[23]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[23]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[23]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[23]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[23]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[23]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[35]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[35]PETSC ERROR: No support for this operation for this object type
[35]PETSC ERROR: No method productsymbolic for Mat of type (null)
[35]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[35]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[35]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[35]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[35]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[35]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[35]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: No support for this operation for this object type
[10]PETSC ERROR: No method productsymbolic for Mat of type (null)
[10]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[10]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[10]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[10]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[10]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[10]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[10]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[5]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[5]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[5]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[5]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[5]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[5]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[5]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[5]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[5]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[5]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: No support for this operation for this object type
[17]PETSC ERROR: No method productsymbolic for Mat of type (null)
[17]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[17]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[17]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[17]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[17]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[17]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[17]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[35]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[10]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[10]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[10]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[5]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[5]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[5]PETSC ERROR: #14 main() at ex13.c:178
[5]PETSC ERROR: PETSc Option Table entries:
[5]PETSC ERROR: -benchmark_it 10
[5]PETSC ERROR: -dm_distribute
[5]PETSC ERROR: -dm_mat_type aijkokkos
[5]PETSC ERROR: -dm_plex_box_faces 4,4,4
[5]PETSC ERROR: -dm_plex_box_lower 0,0,0
[5]PETSC ERROR: -dm_plex_box_upper 2,2,2
[5]PETSC ERROR: -dm_plex_dim 3
[5]PETSC ERROR: -dm_plex_simplex 0
[5]PETSC ERROR: -dm_refine 4
[5]PETSC ERROR: -dm_vec_type kokkos
[5]PETSC ERROR: -dm_view
[5]PETSC ERROR: -ksp_converged_reason
[5]PETSC ERROR: -ksp_max_it 200
[5]PETSC ERROR: -ksp_norm_type unpreconditioned
[5]PETSC ERROR: -ksp_rtol 1.e-12
[5]PETSC ERROR: -ksp_type cg
[5]PETSC ERROR: -log_view
[5]PETSC ERROR: -mat_type aijkokkos
[5]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[5]PETSC ERROR: -mg_levels_ksp_type chebyshev
[36]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[36]PETSC ERROR: No support for this operation for this object type
[36]PETSC ERROR: No method productsymbolic for Mat of type (null)
[36]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[36]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[36]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[36]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[36]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[36]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[36]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[10]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[10]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[10]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[10]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[10]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[10]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[10]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[10]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[10]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[5]PETSC ERROR: -mg_levels_pc_type jacobi
[5]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[5]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[5]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[5]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[5]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[36]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[36]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[36]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: No support for this operation for this object type
[13]PETSC ERROR: No method productsymbolic for Mat of type (null)
[13]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[13]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[13]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[13]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[13]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[13]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[13]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: No support for this operation for this object type
[7]PETSC ERROR: No method productsymbolic for Mat of type (null)
[7]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[7]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[7]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[7]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[7]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[7]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[7]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[7]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001116 by madams Sun Oct 30 05:30:05 2022
[7]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[7]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[7]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[7]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[7]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[7]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[7]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[7]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[7]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[7]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[7]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[7]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[7]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[7]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[7]PETSC ERROR: #14 main() at ex13.c:178
[7]PETSC ERROR: PETSc Option Table entries:
[7]PETSC ERROR: -benchmark_it 10
[7]PETSC ERROR: -dm_distribute
[7]PETSC ERROR: -dm_mat_type aijkokkos
[7]PETSC ERROR: -dm_plex_box_faces 4,4,4
[7]PETSC ERROR: -dm_plex_box_lower 0,0,0
[7]PETSC ERROR: -dm_plex_box_upper 2,2,2
[7]PETSC ERROR: -dm_plex_dim 3
[7]PETSC ERROR: -dm_plex_simplex 0
[7]PETSC ERROR: -dm_refine 4
[7]PETSC ERROR: -dm_vec_type kokkos
[7]PETSC ERROR: -dm_view
[7]PETSC ERROR: -ksp_converged_reason
[7]PETSC ERROR: -ksp_max_it 200
[7]PETSC ERROR: -ksp_norm_type unpreconditioned
[7]PETSC ERROR: -ksp_rtol 1.e-12
[7]PETSC ERROR: -ksp_type cg
[7]PETSC ERROR: -log_view
[7]PETSC ERROR: -mat_type aijkokkos
[7]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[7]PETSC ERROR: -mg_levels_ksp_type chebyshev
[26]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[26]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[26]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[49]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[49]PETSC ERROR: No support for this operation for this object type
[49]PETSC ERROR: No method productsymbolic for Mat of type (null)
[49]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[49]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[49]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[49]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[49]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[49]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[49]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[7]PETSC ERROR: -mg_levels_pc_type jacobi
[7]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[7]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[7]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[7]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[7]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[26]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[26]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[26]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[26]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[26]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[26]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[26]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[26]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[3]PETSC ERROR: -ksp_max_it 200
[3]PETSC ERROR: -ksp_norm_type unpreconditioned
[3]PETSC ERROR: -ksp_rtol 1.e-12
[3]PETSC ERROR: -ksp_type cg
[3]PETSC ERROR: -log_view
[3]PETSC ERROR: -mat_type aijkokkos
[3]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[3]PETSC ERROR: -mg_levels_ksp_type chebyshev
[3]PETSC ERROR: -mg_levels_pc_type jacobi
[3]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[3]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[3]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[3]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[3]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[3]PETSC ERROR: -pc_gamg_process_eq_limit 400
[3]PETSC ERROR: -pc_gamg_repartition false
[3]PETSC ERROR: -pc_gamg_reuse_interpolation true
[3]PETSC ERROR: -pc_gamg_threshold 0.01
[3]PETSC ERROR: -pc_type gamg
[3]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[3]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[3]PETSC ERROR: -petscpartitioner_type simple
[3]PETSC ERROR: -potential_petscspace_degree 2
[3]PETSC ERROR: -snes_lag_jacobian -2
[3]PETSC ERROR: -snes_max_it 1
[3]PETSC ERROR: -snes_rtol 1.e-8
[3]PETSC ERROR: -snes_type ksponly
[3]PETSC ERROR: -use_gpu_aware_mpi 0
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 3] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[5]PETSC ERROR: -pc_gamg_process_eq_limit 400
[5]PETSC ERROR: -pc_gamg_repartition false
[5]PETSC ERROR: -pc_gamg_reuse_interpolation true
[5]PETSC ERROR: -pc_gamg_threshold 0.01
[5]PETSC ERROR: -pc_type gamg
[5]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[5]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[5]PETSC ERROR: -petscpartitioner_type simple
[5]PETSC ERROR: -potential_petscspace_degree 2
[5]PETSC ERROR: -snes_lag_jacobian -2
[5]PETSC ERROR: -snes_max_it 1
[5]PETSC ERROR: -snes_rtol 1.e-8
[5]PETSC ERROR: -snes_type ksponly
[5]PETSC ERROR: -use_gpu_aware_mpi 0
[5]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 5] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[7]PETSC ERROR: -pc_gamg_process_eq_limit 400
[7]PETSC ERROR: -pc_gamg_repartition false
[7]PETSC ERROR: -pc_gamg_reuse_interpolation true
[7]PETSC ERROR: -pc_gamg_threshold 0.01
[7]PETSC ERROR: -pc_type gamg
[7]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[7]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[7]PETSC ERROR: -petscpartitioner_type simple
[7]PETSC ERROR: -potential_petscspace_degree 2
[7]PETSC ERROR: -snes_lag_jacobian -2
[7]PETSC ERROR: -snes_max_it 1
[7]PETSC ERROR: -snes_rtol 1.e-8
[7]PETSC ERROR: -snes_type ksponly
[7]PETSC ERROR: -use_gpu_aware_mpi 0
[7]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 7] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001116] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[59]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[59]PETSC ERROR: No support for this operation for this object type
[59]PETSC ERROR: No method productsymbolic for Mat of type (null)
[59]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[59]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[59]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[59]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[59]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[59]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[59]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[59]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[59]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[59]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[13]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[13]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[13]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[13]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[13]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[13]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[13]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[13]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[13]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[13]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[13]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[13]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[13]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[13]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[13]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[13]PETSC ERROR: #14 main() at ex13.c:178
[13]PETSC ERROR: PETSc Option Table entries:
[13]PETSC ERROR: -benchmark_it 10
[13]PETSC ERROR: -dm_distribute
[13]PETSC ERROR: -dm_mat_type aijkokkos
[13]PETSC ERROR: -dm_plex_box_faces 4,4,4
[13]PETSC ERROR: -dm_plex_box_lower 0,0,0
[13]PETSC ERROR: -dm_plex_box_upper 2,2,2
[13]PETSC ERROR: -dm_plex_dim 3
[13]PETSC ERROR: -dm_plex_simplex 0
[13]PETSC ERROR: -dm_refine 4
[38]PETSC ERROR: PETSc Option Table entries:
[38]PETSC ERROR: -benchmark_it 10
[38]PETSC ERROR: -dm_distribute
[38]PETSC ERROR: -dm_mat_type aijkokkos
[38]PETSC ERROR: -dm_plex_box_faces 4,4,4
[38]PETSC ERROR: -dm_plex_box_lower 0,0,0
[38]PETSC ERROR: -dm_plex_box_upper 2,2,2
[38]PETSC ERROR: -dm_plex_dim 3
[38]PETSC ERROR: -dm_plex_simplex 0
[38]PETSC ERROR: -dm_refine 4
[38]PETSC ERROR: -dm_vec_type kokkos
[38]PETSC ERROR: -dm_view
[38]PETSC ERROR: -ksp_converged_reason
[38]PETSC ERROR: -ksp_max_it 200
[38]PETSC ERROR: -ksp_norm_type unpreconditioned
[38]PETSC ERROR: -ksp_rtol 1.e-12
[38]PETSC ERROR: -ksp_type cg
[38]PETSC ERROR: -log_view
[38]PETSC ERROR: -mat_type aijkokkos
[38]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[38]PETSC ERROR: -mg_levels_ksp_type chebyshev
[38]PETSC ERROR: -mg_levels_pc_type jacobi
[38]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[38]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[38]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: No support for this operation for this object type
[14]PETSC ERROR: No method productsymbolic for Mat of type (null)
[14]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[14]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[14]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[14]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[14]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[14]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[14]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[38]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[38]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[38]PETSC ERROR: -pc_gamg_process_eq_limit 400
[38]PETSC ERROR: -pc_gamg_repartition false
[38]PETSC ERROR: -pc_gamg_reuse_interpolation true
[38]PETSC ERROR: -pc_gamg_threshold 0.01
[38]PETSC ERROR: -pc_type gamg
[38]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[38]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[38]PETSC ERROR: -petscpartitioner_type simple
[38]PETSC ERROR: -potential_petscspace_degree 2
[38]PETSC ERROR: -snes_lag_jacobian -2
[38]PETSC ERROR: -snes_max_it 1
[38]PETSC ERROR: -snes_rtol 1.e-8
[38]PETSC ERROR: -snes_type ksponly
[38]PETSC ERROR: -use_gpu_aware_mpi 0
[38]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[14]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[14]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[14]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[30]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[30]PETSC ERROR: No support for this operation for this object type
[30]PETSC ERROR: No method productsymbolic for Mat of type (null)
[30]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[30]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[30]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[30]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[30]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[30]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[30]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[30]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[30]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[30]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[49]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[49]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[49]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[44]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[44]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[44]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[44]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[44]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[44]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[44]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[44]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[44]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[30]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[30]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[30]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[30]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[30]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[30]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[30]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[30]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[30]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[51]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[51]PETSC ERROR: No support for this operation for this object type
[51]PETSC ERROR: No method productsymbolic for Mat of type (null)
[51]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[51]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[51]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[51]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[51]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[51]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[51]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[44]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[51]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[51]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[51]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[46]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[46]PETSC ERROR: No support for this operation for this object type
[46]PETSC ERROR: No method productsymbolic for Mat of type (null)
[46]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[46]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[46]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[46]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[46]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[46]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[46]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[52]PETSC ERROR: -dm_plex_box_faces 4,4,4
[52]PETSC ERROR: -dm_plex_box_lower 0,0,0
[52]PETSC ERROR: -dm_plex_box_upper 2,2,2
[52]PETSC ERROR: -dm_plex_dim 3
[52]PETSC ERROR: -dm_plex_simplex 0
[52]PETSC ERROR: -dm_refine 4
[52]PETSC ERROR: -dm_vec_type kokkos
[52]PETSC ERROR: -dm_view
[52]PETSC ERROR: -ksp_converged_reason
[52]PETSC ERROR: -ksp_max_it 200
[52]PETSC ERROR: -ksp_norm_type unpreconditioned
[52]PETSC ERROR: -ksp_rtol 1.e-12
[52]PETSC ERROR: -ksp_type cg
[52]PETSC ERROR: -log_view
[52]PETSC ERROR: -mat_type aijkokkos
[52]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[52]PETSC ERROR: -mg_levels_ksp_type chebyshev
[52]PETSC ERROR: -mg_levels_pc_type jacobi
[52]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[52]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[52]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[52]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[52]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[52]PETSC ERROR: -pc_gamg_process_eq_limit 400
[46]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[46]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[46]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[61]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[61]PETSC ERROR: No support for this operation for this object type
[61]PETSC ERROR: No method productsymbolic for Mat of type (null)
[61]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[61]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[61]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[61]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[61]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[61]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[61]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[14]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[14]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[14]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[14]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[14]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[14]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[14]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[14]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[14]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[14]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[14]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[14]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[14]PETSC ERROR: #14 main() at ex13.c:178
[14]PETSC ERROR: PETSc Option Table entries:
[14]PETSC ERROR: -benchmark_it 10
[14]PETSC ERROR: -dm_distribute
[14]PETSC ERROR: -dm_mat_type aijkokkos
[14]PETSC ERROR: -dm_plex_box_faces 4,4,4
[14]PETSC ERROR: -dm_plex_box_lower 0,0,0
[14]PETSC ERROR: -dm_plex_box_upper 2,2,2
[14]PETSC ERROR: -dm_plex_dim 3
[14]PETSC ERROR: -dm_plex_simplex 0
[14]PETSC ERROR: -dm_refine 4
[14]PETSC ERROR: -dm_vec_type kokkos
[14]PETSC ERROR: -dm_view
[14]PETSC ERROR: -ksp_converged_reason
[14]PETSC ERROR: -ksp_max_it 200
[14]PETSC ERROR: -ksp_norm_type unpreconditioned
[14]PETSC ERROR: -ksp_rtol 1.e-12
[14]PETSC ERROR: -ksp_type cg
[14]PETSC ERROR: -log_view
[14]PETSC ERROR: -mat_type aijkokkos
[14]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[14]PETSC ERROR: -mg_levels_ksp_type chebyshev
[14]PETSC ERROR: -mg_levels_pc_type jacobi
[14]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[14]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[14]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[14]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[8]PETSC ERROR: -pc_gamg_repartition false
[8]PETSC ERROR: -pc_gamg_reuse_interpolation true
[8]PETSC ERROR: -pc_gamg_threshold 0.01
[8]PETSC ERROR: -pc_type gamg
[8]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[8]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[8]PETSC ERROR: -petscpartitioner_type simple
[8]PETSC ERROR: -potential_petscspace_degree 2
[8]PETSC ERROR: -snes_lag_jacobian -2
[8]PETSC ERROR: -snes_max_it 1
[8]PETSC ERROR: -snes_rtol 1.e-8
[8]PETSC ERROR: -snes_type ksponly
[8]PETSC ERROR: -use_gpu_aware_mpi 0
[8]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 8] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[9]PETSC ERROR: #14 main() at ex13.c:178
[9]PETSC ERROR: PETSc Option Table entries:
[9]PETSC ERROR: -benchmark_it 10
[9]PETSC ERROR: -dm_distribute
[9]PETSC ERROR: -dm_mat_type aijkokkos
[9]PETSC ERROR: -dm_plex_box_faces 4,4,4
[9]PETSC ERROR: -dm_plex_box_lower 0,0,0
[9]PETSC ERROR: -dm_plex_box_upper 2,2,2
[9]PETSC ERROR: -dm_plex_dim 3
[9]PETSC ERROR: -dm_plex_simplex 0
[9]PETSC ERROR: -dm_refine 4
[9]PETSC ERROR: -dm_vec_type kokkos
[9]PETSC ERROR: -dm_view
[9]PETSC ERROR: -ksp_converged_reason
[9]PETSC ERROR: -ksp_max_it 200
[9]PETSC ERROR: -ksp_norm_type unpreconditioned
[9]PETSC ERROR: -ksp_rtol 1.e-12
[9]PETSC ERROR: -ksp_type cg
[9]PETSC ERROR: -log_view
[9]PETSC ERROR: -mat_type aijkokkos
[9]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[9]PETSC ERROR: -mg_levels_ksp_type chebyshev
[9]PETSC ERROR: -mg_levels_pc_type jacobi
[9]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[9]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[9]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[17]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[17]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[17]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[17]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[17]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[39]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[39]PETSC ERROR: No support for this operation for this object type
[39]PETSC ERROR: No method productsymbolic for Mat of type (null)
[39]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[39]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[39]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[39]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[39]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[39]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[39]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: No support for this operation for this object type
[19]PETSC ERROR: No method productsymbolic for Mat of type (null)
[19]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[19]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[19]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[19]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[19]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[19]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[19]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[39]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[39]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[39]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[19]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[19]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[19]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[39]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[39]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[39]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[39]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[39]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[39]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[39]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[39]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[39]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[39]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[39]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[39]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[39]PETSC ERROR: #14 main() at ex13.c:178
[39]PETSC ERROR: PETSc Option Table entries:
[39]PETSC ERROR: -benchmark_it 10
[39]PETSC ERROR: -dm_distribute
[30]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[30]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[30]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[30]PETSC ERROR: #14 main() at ex13.c:178
[30]PETSC ERROR: PETSc Option Table entries:
[30]PETSC ERROR: -benchmark_it 10
[52]PETSC ERROR: -pc_gamg_repartition false
[52]PETSC ERROR: -pc_gamg_reuse_interpolation true
[52]PETSC ERROR: -pc_gamg_threshold 0.01
[52]PETSC ERROR: -pc_type gamg
[52]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[52]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[52]PETSC ERROR: -petscpartitioner_type simple
[52]PETSC ERROR: -potential_petscspace_degree 2
[52]PETSC ERROR: -snes_lag_jacobian -2
[52]PETSC ERROR: -snes_max_it 1
[52]PETSC ERROR: -snes_rtol 1.e-8
[52]PETSC ERROR: -snes_type ksponly
[52]PETSC ERROR: -use_gpu_aware_mpi 0
[52]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: No support for this operation for this object type
[24]PETSC ERROR: No method productsymbolic for Mat of type (null)
[24]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[24]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[24]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[24]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[24]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[24]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[24]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[61]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[61]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[61]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[9]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[9]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[9]PETSC ERROR: -pc_gamg_process_eq_limit 400
[9]PETSC ERROR: -pc_gamg_repartition false
[9]PETSC ERROR: -pc_gamg_reuse_interpolation true
[9]PETSC ERROR: -pc_gamg_threshold 0.01
[9]PETSC ERROR: -pc_type gamg
[9]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[9]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[9]PETSC ERROR: -petscpartitioner_type simple
[9]PETSC ERROR: -potential_petscspace_degree 2
[9]PETSC ERROR: -snes_lag_jacobian -2
[9]PETSC ERROR: -snes_max_it 1
[9]PETSC ERROR: -snes_rtol 1.e-8
[9]PETSC ERROR: -snes_type ksponly
[9]PETSC ERROR: -use_gpu_aware_mpi 0
[9]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 9] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[61]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[61]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[10]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[10]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[10]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[10]PETSC ERROR: #14 main() at ex13.c:178
[10]PETSC ERROR: PETSc Option Table entries:
[10]PETSC ERROR: -benchmark_it 10
[10]PETSC ERROR: -dm_distribute
[10]PETSC ERROR: -dm_mat_type aijkokkos
[10]PETSC ERROR: -dm_plex_box_faces 4,4,4
[10]PETSC ERROR: -dm_plex_box_lower 0,0,0
[10]PETSC ERROR: -dm_plex_box_upper 2,2,2
[10]PETSC ERROR: -dm_plex_dim 3
[10]PETSC ERROR: -dm_plex_simplex 0
[10]PETSC ERROR: -dm_refine 4
[10]PETSC ERROR: -dm_vec_type kokkos
[10]PETSC ERROR: -dm_view
[10]PETSC ERROR: -ksp_converged_reason
[10]PETSC ERROR: -ksp_max_it 200
[10]PETSC ERROR: -ksp_norm_type unpreconditioned
[10]PETSC ERROR: -ksp_rtol 1.e-12
[10]PETSC ERROR: -ksp_type cg
[10]PETSC ERROR: -log_view
[62]PETSC ERROR: -dm_distribute
[62]PETSC ERROR: -dm_mat_type aijkokkos
[62]PETSC ERROR: -dm_plex_box_faces 4,4,4
[62]PETSC ERROR: -dm_plex_box_lower 0,0,0
[62]PETSC ERROR: -dm_plex_box_upper 2,2,2
[62]PETSC ERROR: -dm_plex_dim 3
[62]PETSC ERROR: -dm_plex_simplex 0
[62]PETSC ERROR: -dm_refine 4
[62]PETSC ERROR: -dm_vec_type kokkos
[62]PETSC ERROR: -dm_view
[62]PETSC ERROR: -ksp_converged_reason
[62]PETSC ERROR: -ksp_max_it 200
[62]PETSC ERROR: -ksp_norm_type unpreconditioned
[62]PETSC ERROR: -ksp_rtol 1.e-12
[62]PETSC ERROR: -ksp_type cg
[62]PETSC ERROR: -log_view
[62]PETSC ERROR: -mat_type aijkokkos
[62]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[62]PETSC ERROR: -mg_levels_ksp_type chebyshev
[62]PETSC ERROR: -mg_levels_pc_type jacobi
[62]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[62]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[62]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[62]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[62]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[10]PETSC ERROR: -mat_type aijkokkos
[10]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[10]PETSC ERROR: -mg_levels_ksp_type chebyshev
[10]PETSC ERROR: -mg_levels_pc_type jacobi
[10]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[10]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[10]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[10]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[10]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[10]PETSC ERROR: -pc_gamg_process_eq_limit 400
[10]PETSC ERROR: -pc_gamg_repartition false
[10]PETSC ERROR: -pc_gamg_reuse_interpolation true
[10]PETSC ERROR: -pc_gamg_threshold 0.01
[10]PETSC ERROR: -pc_type gamg
[10]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[10]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[10]PETSC ERROR: -petscpartitioner_type simple
[10]PETSC ERROR: -potential_petscspace_degree 2
[10]PETSC ERROR: -snes_lag_jacobian -2
[10]PETSC ERROR: -snes_max_it 1
[10]PETSC ERROR: -snes_rtol 1.e-8
[10]PETSC ERROR: -snes_type ksponly
[62]PETSC ERROR: -pc_gamg_process_eq_limit 400
[62]PETSC ERROR: -pc_gamg_repartition false
[62]PETSC ERROR: -pc_gamg_reuse_interpolation true
[62]PETSC ERROR: -pc_gamg_threshold 0.01
[62]PETSC ERROR: -pc_type gamg
[62]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[62]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[62]PETSC ERROR: -petscpartitioner_type simple
[62]PETSC ERROR: -potential_petscspace_degree 2
[62]PETSC ERROR: -snes_lag_jacobian -2
[62]PETSC ERROR: -snes_max_it 1
[62]PETSC ERROR: -snes_rtol 1.e-8
[62]PETSC ERROR: -snes_type ksponly
[62]PETSC ERROR: -use_gpu_aware_mpi 0
[62]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 62] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[10]PETSC ERROR: -use_gpu_aware_mpi 0
[10]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 10] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: No support for this operation for this object type
[11]PETSC ERROR: No method productsymbolic for Mat of type (null)
[11]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[11]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[11]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[11]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[11]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[11]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[11]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[19]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[19]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[19]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[19]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[19]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[19]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[19]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[19]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[19]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[11]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[11]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[11]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[19]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[19]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[19]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[19]PETSC ERROR: #14 main() at ex13.c:178
[11]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[11]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[11]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[11]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[11]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[11]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[11]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[11]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[11]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: No support for this operation for this object type
[20]PETSC ERROR: No method productsymbolic for Mat of type (null)
[20]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[20]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[20]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[20]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[20]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[20]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[20]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[53]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[53]PETSC ERROR: No support for this operation for this object type
[53]PETSC ERROR: No method productsymbolic for Mat of type (null)
[53]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[53]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[53]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[53]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[53]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[53]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[53]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[46]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[46]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[46]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[46]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[46]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[46]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[46]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[46]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[46]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[11]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[11]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[11]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[11]PETSC ERROR: #14 main() at ex13.c:178
[11]PETSC ERROR: PETSc Option Table entries:
[11]PETSC ERROR: -benchmark_it 10
[11]PETSC ERROR: -dm_distribute
[11]PETSC ERROR: -dm_mat_type aijkokkos
[11]PETSC ERROR: -dm_plex_box_faces 4,4,4
[53]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[53]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[53]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[46]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: No support for this operation for this object type
[12]PETSC ERROR: No method productsymbolic for Mat of type (null)
[12]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[12]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[12]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[12]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[12]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[12]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[12]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[53]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[53]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[53]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[53]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[53]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[53]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[53]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[53]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[53]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[40]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[40]PETSC ERROR: No support for this operation for this object type
[40]PETSC ERROR: No method productsymbolic for Mat of type (null)
[40]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[40]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[40]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[40]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[40]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[40]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[40]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[12]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[12]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[12]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[53]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[53]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[53]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[53]PETSC ERROR: #14 main() at ex13.c:178
[53]PETSC ERROR: PETSc Option Table entries:
[53]PETSC ERROR: -benchmark_it 10
[53]PETSC ERROR: -dm_distribute
[53]PETSC ERROR: -dm_mat_type aijkokkos
[53]PETSC ERROR: -dm_plex_box_faces 4,4,4
[53]PETSC ERROR: -dm_plex_box_lower 0,0,0
[53]PETSC ERROR: -dm_plex_box_upper 2,2,2
[53]PETSC ERROR: -dm_plex_dim 3
[53]PETSC ERROR: -dm_plex_simplex 0
[53]PETSC ERROR: -dm_refine 4
[53]PETSC ERROR: -dm_vec_type kokkos
[53]PETSC ERROR: -dm_view
[12]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[12]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[12]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[12]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[12]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[12]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[12]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[12]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[12]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[54]PETSC ERROR: -log_view
[54]PETSC ERROR: -mat_type aijkokkos
[54]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[54]PETSC ERROR: -mg_levels_ksp_type chebyshev
[54]PETSC ERROR: -mg_levels_pc_type jacobi
[54]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[54]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[54]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[54]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[54]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[54]PETSC ERROR: -pc_gamg_process_eq_limit 400
[54]PETSC ERROR: -pc_gamg_repartition false
[54]PETSC ERROR: -pc_gamg_reuse_interpolation true
[54]PETSC ERROR: -pc_gamg_threshold 0.01
[54]PETSC ERROR: -pc_type gamg
[54]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[54]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[54]PETSC ERROR: -petscpartitioner_type simple
[54]PETSC ERROR: -potential_petscspace_degree 2
[54]PETSC ERROR: -snes_lag_jacobian -2
[54]PETSC ERROR: -snes_max_it 1
[54]PETSC ERROR: -snes_rtol 1.e-8
[54]PETSC ERROR: -snes_type ksponly
[54]PETSC ERROR: -use_gpu_aware_mpi 0
[54]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 54] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[55]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[55]PETSC ERROR: No support for this operation for this object type
[55]PETSC ERROR: No method productsymbolic for Mat of type (null)
[55]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[55]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[55]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[55]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[55]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[55]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[55]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[63]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[63]PETSC ERROR: No support for this operation for this object type
[63]PETSC ERROR: No method productsymbolic for Mat of type (null)
[63]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[63]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[63]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[63]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[63]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[63]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[63]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[63]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[63]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[63]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[12]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[12]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[12]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[12]PETSC ERROR: #14 main() at ex13.c:178
[12]PETSC ERROR: PETSc Option Table entries:
[12]PETSC ERROR: -benchmark_it 10
[12]PETSC ERROR: -dm_distribute
[12]PETSC ERROR: -dm_mat_type aijkokkos
[12]PETSC ERROR: -dm_plex_box_faces 4,4,4
[12]PETSC ERROR: -dm_plex_box_lower 0,0,0
[12]PETSC ERROR: -dm_plex_box_upper 2,2,2
[12]PETSC ERROR: -dm_plex_dim 3
[12]PETSC ERROR: -dm_plex_simplex 0
[12]PETSC ERROR: -dm_refine 4
[12]PETSC ERROR: -dm_vec_type kokkos
[12]PETSC ERROR: -dm_view
[12]PETSC ERROR: -ksp_converged_reason
[12]PETSC ERROR: -ksp_max_it 200
[12]PETSC ERROR: -ksp_norm_type unpreconditioned
[12]PETSC ERROR: -ksp_rtol 1.e-12
[12]PETSC ERROR: -ksp_type cg
[12]PETSC ERROR: -log_view
[63]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[63]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[63]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[63]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[63]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[63]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[63]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[63]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[63]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[12]PETSC ERROR: -mat_type aijkokkos
[63]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[63]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[63]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[63]PETSC ERROR: #14 main() at ex13.c:178
[63]PETSC ERROR: PETSc Option Table entries:
[63]PETSC ERROR: -benchmark_it 10
[63]PETSC ERROR: -dm_distribute
[63]PETSC ERROR: -dm_mat_type aijkokkos
[63]PETSC ERROR: -dm_plex_box_faces 4,4,4
[63]PETSC ERROR: -dm_plex_box_lower 0,0,0
[63]PETSC ERROR: -dm_plex_box_upper 2,2,2
[63]PETSC ERROR: -dm_plex_dim 3
[63]PETSC ERROR: -dm_plex_simplex 0
[63]PETSC ERROR: -dm_refine 4
[63]PETSC ERROR: -dm_vec_type kokkos
[63]PETSC ERROR: -dm_view
[63]PETSC ERROR: -ksp_converged_reason
[13]PETSC ERROR: -dm_vec_type kokkos
[13]PETSC ERROR: -dm_view
[13]PETSC ERROR: -ksp_converged_reason
[13]PETSC ERROR: -ksp_max_it 200
[13]PETSC ERROR: -ksp_norm_type unpreconditioned
[13]PETSC ERROR: -ksp_rtol 1.e-12
[13]PETSC ERROR: -ksp_type cg
[13]PETSC ERROR: -log_view
[13]PETSC ERROR: -mat_type aijkokkos
[13]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[13]PETSC ERROR: -mg_levels_ksp_type chebyshev
[13]PETSC ERROR: -mg_levels_pc_type jacobi
[13]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[13]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[13]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[13]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[13]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[13]PETSC ERROR: -pc_gamg_process_eq_limit 400
[13]PETSC ERROR: -pc_gamg_repartition false
[13]PETSC ERROR: -pc_gamg_reuse_interpolation true
[13]PETSC ERROR: -pc_gamg_threshold 0.01
[13]PETSC ERROR: -pc_type gamg
[13]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[57]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[57]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[57]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[13]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[13]PETSC ERROR: -petscpartitioner_type simple
[13]PETSC ERROR: -potential_petscspace_degree 2
[13]PETSC ERROR: -snes_lag_jacobian -2
[13]PETSC ERROR: -snes_max_it 1
[13]PETSC ERROR: -snes_rtol 1.e-8
[13]PETSC ERROR: -snes_type ksponly
[13]PETSC ERROR: -use_gpu_aware_mpi 0
[13]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 13] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[57]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[57]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[57]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[57]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[57]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[57]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[57]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[57]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[57]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[14]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[14]PETSC ERROR: -pc_gamg_process_eq_limit 400
[14]PETSC ERROR: -pc_gamg_repartition false
[14]PETSC ERROR: -pc_gamg_reuse_interpolation true
[14]PETSC ERROR: -pc_gamg_threshold 0.01
[14]PETSC ERROR: -pc_type gamg
[14]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[14]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[14]PETSC ERROR: -petscpartitioner_type simple
[14]PETSC ERROR: -potential_petscspace_degree 2
[14]PETSC ERROR: -snes_lag_jacobian -2
[14]PETSC ERROR: -snes_max_it 1
[14]PETSC ERROR: -snes_rtol 1.e-8
[14]PETSC ERROR: -snes_type ksponly
[14]PETSC ERROR: -use_gpu_aware_mpi 0
[14]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 14] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[57]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[57]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[57]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[57]PETSC ERROR: #14 main() at ex13.c:178
[57]PETSC ERROR: PETSc Option Table entries:
[57]PETSC ERROR: -benchmark_it 10
[57]PETSC ERROR: -dm_distribute
[57]PETSC ERROR: -dm_mat_type aijkokkos
[57]PETSC ERROR: -dm_plex_box_faces 4,4,4
[57]PETSC ERROR: -dm_plex_box_lower 0,0,0
[57]PETSC ERROR: -dm_plex_box_upper 2,2,2
[57]PETSC ERROR: -dm_plex_dim 3
[57]PETSC ERROR: -dm_plex_simplex 0
[57]PETSC ERROR: -dm_refine 4
[57]PETSC ERROR: -dm_vec_type kokkos
[57]PETSC ERROR: -dm_view
[57]PETSC ERROR: -ksp_converged_reason
[57]PETSC ERROR: -ksp_max_it 200
[57]PETSC ERROR: -ksp_norm_type unpreconditioned
[57]PETSC ERROR: -ksp_rtol 1.e-12
[57]PETSC ERROR: -ksp_type cg
[57]PETSC ERROR: -log_view
[15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: No support for this operation for this object type
[15]PETSC ERROR: No method productsymbolic for Mat of type (null)
[15]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[15]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[15]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[15]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[15]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[15]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[15]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[59]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[59]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[59]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[59]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[59]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[59]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[59]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[59]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[59]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[15]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001117 by madams Sun Oct 30 05:30:05 2022
[15]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[15]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[55]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[55]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[55]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[55]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[55]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[55]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[55]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[55]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[55]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[55]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[55]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[55]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[40]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[40]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[40]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[41]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[41]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[41]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[41]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[41]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[41]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[41]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[41]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[41]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[41]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[41]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[41]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[41]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[41]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[59]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[59]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[59]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[59]PETSC ERROR: #14 main() at ex13.c:178
[59]PETSC ERROR: PETSc Option Table entries:
[59]PETSC ERROR: -benchmark_it 10
[59]PETSC ERROR: -dm_distribute
[59]PETSC ERROR: -dm_mat_type aijkokkos
[59]PETSC ERROR: -dm_plex_box_faces 4,4,4
[59]PETSC ERROR: -dm_plex_box_lower 0,0,0
[59]PETSC ERROR: -dm_plex_box_upper 2,2,2
[59]PETSC ERROR: -dm_plex_dim 3
[59]PETSC ERROR: -dm_plex_simplex 0
[59]PETSC ERROR: -dm_refine 4
[59]PETSC ERROR: -dm_vec_type kokkos
[59]PETSC ERROR: -dm_view
[59]PETSC ERROR: -ksp_converged_reason
[59]PETSC ERROR: -ksp_max_it 200
[59]PETSC ERROR: -ksp_norm_type unpreconditioned
[59]PETSC ERROR: -ksp_rtol 1.e-12
[59]PETSC ERROR: -ksp_type cg
[59]PETSC ERROR: -log_view
[15]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[15]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[15]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[15]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[15]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[15]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[15]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[15]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[15]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[59]PETSC ERROR: -mat_type aijkokkos
[59]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[59]PETSC ERROR: -mg_levels_ksp_type chebyshev
[59]PETSC ERROR: -mg_levels_pc_type jacobi
[59]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[59]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[59]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[59]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[59]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[59]PETSC ERROR: -pc_gamg_process_eq_limit 400
[59]PETSC ERROR: -pc_gamg_repartition false
[59]PETSC ERROR: -pc_gamg_reuse_interpolation true
[59]PETSC ERROR: -pc_gamg_threshold 0.01
[59]PETSC ERROR: -pc_type gamg
[59]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[59]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[59]PETSC ERROR: -petscpartitioner_type simple
[59]PETSC ERROR: -potential_petscspace_degree 2
[59]PETSC ERROR: -snes_lag_jacobian -2
[59]PETSC ERROR: -snes_max_it 1
[59]PETSC ERROR: -snes_rtol 1.e-8
[59]PETSC ERROR: -snes_type ksponly
[15]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[15]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[15]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[15]PETSC ERROR: #14 main() at ex13.c:178
[15]PETSC ERROR: PETSc Option Table entries:
[15]PETSC ERROR: -benchmark_it 10
[15]PETSC ERROR: -dm_distribute
[15]PETSC ERROR: -dm_mat_type aijkokkos
[15]PETSC ERROR: -dm_plex_box_faces 4,4,4
[15]PETSC ERROR: -dm_plex_box_lower 0,0,0
[15]PETSC ERROR: -dm_plex_box_upper 2,2,2
[15]PETSC ERROR: -dm_plex_dim 3
[15]PETSC ERROR: -dm_plex_simplex 0
[15]PETSC ERROR: -dm_refine 4
[15]PETSC ERROR: -dm_vec_type kokkos
[15]PETSC ERROR: -dm_view
[15]PETSC ERROR: -ksp_converged_reason
[15]PETSC ERROR: -ksp_max_it 200
[15]PETSC ERROR: -ksp_norm_type unpreconditioned
[15]PETSC ERROR: -ksp_rtol 1.e-12
[15]PETSC ERROR: -ksp_type cg
[15]PETSC ERROR: -log_view
[20]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[20]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[20]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[15]PETSC ERROR: -mat_type aijkokkos
[15]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[15]PETSC ERROR: -mg_levels_ksp_type chebyshev
[15]PETSC ERROR: -mg_levels_pc_type jacobi
[15]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[15]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[15]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[15]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[11]PETSC ERROR: -dm_plex_box_lower 0,0,0
[11]PETSC ERROR: -dm_plex_box_upper 2,2,2
[11]PETSC ERROR: -dm_plex_dim 3
[11]PETSC ERROR: -dm_plex_simplex 0
[11]PETSC ERROR: -dm_refine 4
[11]PETSC ERROR: -dm_vec_type kokkos
[11]PETSC ERROR: -dm_view
[11]PETSC ERROR: -ksp_converged_reason
[11]PETSC ERROR: -ksp_max_it 200
[11]PETSC ERROR: -ksp_norm_type unpreconditioned
[11]PETSC ERROR: -ksp_rtol 1.e-12
[11]PETSC ERROR: -ksp_type cg
[11]PETSC ERROR: -log_view
[11]PETSC ERROR: -mat_type aijkokkos
[11]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[11]PETSC ERROR: -mg_levels_ksp_type chebyshev
[11]PETSC ERROR: -mg_levels_pc_type jacobi
[11]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[11]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[11]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[11]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[11]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[11]PETSC ERROR: -pc_gamg_process_eq_limit 400
[11]PETSC ERROR: -pc_gamg_repartition false
[11]PETSC ERROR: -pc_gamg_reuse_interpolation true
[11]PETSC ERROR: -pc_gamg_threshold 0.01
[11]PETSC ERROR: -pc_type gamg
[11]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[11]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[11]PETSC ERROR: -petscpartitioner_type simple
[11]PETSC ERROR: -potential_petscspace_degree 2
[11]PETSC ERROR: -snes_lag_jacobian -2
[11]PETSC ERROR: -snes_max_it 1
[11]PETSC ERROR: -snes_rtol 1.e-8
[11]PETSC ERROR: -snes_type ksponly
[11]PETSC ERROR: -use_gpu_aware_mpi 0
[11]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 11] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[24]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[24]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[24]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[55]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[55]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[55]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[55]PETSC ERROR: #14 main() at ex13.c:178
[55]PETSC ERROR: PETSc Option Table entries:
[55]PETSC ERROR: -benchmark_it 10
[55]PETSC ERROR: -dm_distribute
[55]PETSC ERROR: -dm_mat_type aijkokkos
[55]PETSC ERROR: -dm_plex_box_faces 4,4,4
[55]PETSC ERROR: -dm_plex_box_lower 0,0,0
[55]PETSC ERROR: -dm_plex_box_upper 2,2,2
[55]PETSC ERROR: -dm_plex_dim 3
[55]PETSC ERROR: -dm_plex_simplex 0
[55]PETSC ERROR: -dm_refine 4
[55]PETSC ERROR: -dm_vec_type kokkos
[55]PETSC ERROR: -dm_view
[55]PETSC ERROR: -ksp_converged_reason
[55]PETSC ERROR: -ksp_max_it 200
[55]PETSC ERROR: -ksp_norm_type unpreconditioned
[55]PETSC ERROR: -ksp_rtol 1.e-12
[55]PETSC ERROR: -ksp_type cg
[55]PETSC ERROR: -log_view
[41]PETSC ERROR: #14 main() at ex13.c:178
[41]PETSC ERROR: PETSc Option Table entries:
[41]PETSC ERROR: -benchmark_it 10
[41]PETSC ERROR: -dm_distribute
[41]PETSC ERROR: -dm_mat_type aijkokkos
[41]PETSC ERROR: -dm_plex_box_faces 4,4,4
[41]PETSC ERROR: -dm_plex_box_lower 0,0,0
[41]PETSC ERROR: -dm_plex_box_upper 2,2,2
[41]PETSC ERROR: -dm_plex_dim 3
[41]PETSC ERROR: -dm_plex_simplex 0
[41]PETSC ERROR: -dm_refine 4
[41]PETSC ERROR: -dm_vec_type kokkos
[41]PETSC ERROR: -dm_view
[41]PETSC ERROR: -ksp_converged_reason
[41]PETSC ERROR: -ksp_max_it 200
[41]PETSC ERROR: -ksp_norm_type unpreconditioned
[41]PETSC ERROR: -ksp_rtol 1.e-12
[41]PETSC ERROR: -ksp_type cg
[41]PETSC ERROR: -log_view
[41]PETSC ERROR: -mat_type aijkokkos
[41]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[41]PETSC ERROR: -mg_levels_ksp_type chebyshev
[41]PETSC ERROR: -mg_levels_pc_type jacobi
[41]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[41]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[24]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[24]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[24]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[24]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[55]PETSC ERROR: -mat_type aijkokkos
[55]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[55]PETSC ERROR: -mg_levels_ksp_type chebyshev
[55]PETSC ERROR: -mg_levels_pc_type jacobi
[55]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[55]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[55]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[55]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[41]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[41]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[41]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[41]PETSC ERROR: -pc_gamg_process_eq_limit 400
[41]PETSC ERROR: -pc_gamg_repartition false
[41]PETSC ERROR: -pc_gamg_reuse_interpolation true
[41]PETSC ERROR: -pc_gamg_threshold 0.01
[41]PETSC ERROR: -pc_type gamg
[41]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[41]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[25]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[25]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[25]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[25]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[42]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[42]PETSC ERROR: No support for this operation for this object type
[42]PETSC ERROR: No method productsymbolic for Mat of type (null)
[42]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[42]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[42]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[42]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[42]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[42]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[42]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[33]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[33]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[33]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[33]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[33]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[20]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[20]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[20]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[20]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[20]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[20]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[20]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[20]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[20]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[20]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[20]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[20]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[20]PETSC ERROR: #14 main() at ex13.c:178
[20]PETSC ERROR: PETSc Option Table entries:
[20]PETSC ERROR: -benchmark_it 10
[20]PETSC ERROR: -dm_distribute
[20]PETSC ERROR: -dm_mat_type aijkokkos
[20]PETSC ERROR: -dm_plex_box_faces 4,4,4
[20]PETSC ERROR: -dm_plex_box_lower 0,0,0
[20]PETSC ERROR: -dm_plex_box_upper 2,2,2
[20]PETSC ERROR: -dm_plex_dim 3
[20]PETSC ERROR: -dm_plex_simplex 0
[20]PETSC ERROR: -dm_refine 4
[20]PETSC ERROR: -dm_vec_type kokkos
[20]PETSC ERROR: -dm_view
[20]PETSC ERROR: -ksp_converged_reason
[33]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[33]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[33]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[33]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[33]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[33]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[33]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[33]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[33]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[33]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[33]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[33]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[33]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[33]PETSC ERROR: #14 main() at ex13.c:178
[33]PETSC ERROR: PETSc Option Table entries:
[33]PETSC ERROR: -benchmark_it 10
[33]PETSC ERROR: -dm_distribute
[33]PETSC ERROR: -dm_mat_type aijkokkos
[33]PETSC ERROR: -dm_plex_box_faces 4,4,4
[33]PETSC ERROR: -dm_plex_box_lower 0,0,0
[33]PETSC ERROR: -dm_plex_box_upper 2,2,2
[33]PETSC ERROR: -dm_plex_dim 3
[33]PETSC ERROR: -dm_plex_simplex 0
[33]PETSC ERROR: -dm_refine 4
[33]PETSC ERROR: -dm_vec_type kokkos
[33]PETSC ERROR: -dm_view
[33]PETSC ERROR: -ksp_converged_reason
[33]PETSC ERROR: -ksp_max_it 200
[35]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[35]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[35]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[35]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[25]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[25]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[25]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[25]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[25]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[25]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[25]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[25]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[25]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[25]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[49]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[49]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[49]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[49]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[49]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[49]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[49]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[49]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[49]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[25]PETSC ERROR: #14 main() at ex13.c:178
[25]PETSC ERROR: PETSc Option Table entries:
[25]PETSC ERROR: -benchmark_it 10
[25]PETSC ERROR: -dm_distribute
[25]PETSC ERROR: -dm_mat_type aijkokkos
[25]PETSC ERROR: -dm_plex_box_faces 4,4,4
[25]PETSC ERROR: -dm_plex_box_lower 0,0,0
[25]PETSC ERROR: -dm_plex_box_upper 2,2,2
[25]PETSC ERROR: -dm_plex_dim 3
[25]PETSC ERROR: -dm_plex_simplex 0
[25]PETSC ERROR: -dm_refine 4
[25]PETSC ERROR: -dm_vec_type kokkos
[25]PETSC ERROR: -dm_view
[25]PETSC ERROR: -ksp_converged_reason
[25]PETSC ERROR: -ksp_max_it 200
[25]PETSC ERROR: -ksp_norm_type unpreconditioned
[25]PETSC ERROR: -ksp_rtol 1.e-12
[25]PETSC ERROR: -ksp_type cg
[25]PETSC ERROR: -log_view
[25]PETSC ERROR: -mat_type aijkokkos
[25]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[25]PETSC ERROR: -mg_levels_ksp_type chebyshev
[25]PETSC ERROR: -mg_levels_pc_type jacobi
[25]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[25]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[49]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[49]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[49]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[49]PETSC ERROR: #14 main() at ex13.c:178
[49]PETSC ERROR: PETSc Option Table entries:
[49]PETSC ERROR: -benchmark_it 10
[49]PETSC ERROR: -dm_distribute
[49]PETSC ERROR: -dm_mat_type aijkokkos
[49]PETSC ERROR: -dm_plex_box_faces 4,4,4
[49]PETSC ERROR: -dm_plex_box_lower 0,0,0
[49]PETSC ERROR: -dm_plex_box_upper 2,2,2
[49]PETSC ERROR: -dm_plex_dim 3
[49]PETSC ERROR: -dm_plex_simplex 0
[49]PETSC ERROR: -dm_refine 4
[49]PETSC ERROR: -dm_vec_type kokkos
[49]PETSC ERROR: -dm_view
[49]PETSC ERROR: -ksp_converged_reason
[49]PETSC ERROR: -ksp_max_it 200
[49]PETSC ERROR: -ksp_norm_type unpreconditioned
[49]PETSC ERROR: -ksp_rtol 1.e-12
[49]PETSC ERROR: -ksp_type cg
[49]PETSC ERROR: -log_view
[42]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[42]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[42]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[25]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[25]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[25]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[25]PETSC ERROR: -pc_gamg_process_eq_limit 400
[25]PETSC ERROR: -pc_gamg_repartition false
[25]PETSC ERROR: -pc_gamg_reuse_interpolation true
[25]PETSC ERROR: -pc_gamg_threshold 0.01
[25]PETSC ERROR: -pc_type gamg
[25]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[25]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[25]PETSC ERROR: -petscpartitioner_type simple
[25]PETSC ERROR: -potential_petscspace_degree 2
[25]PETSC ERROR: -snes_lag_jacobian -2
[25]PETSC ERROR: -snes_max_it 1
[25]PETSC ERROR: -snes_rtol 1.e-8
[25]PETSC ERROR: -snes_type ksponly
[25]PETSC ERROR: -use_gpu_aware_mpi 0
[49]PETSC ERROR: -mat_type aijkokkos
[49]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[49]PETSC ERROR: -mg_levels_ksp_type chebyshev
[49]PETSC ERROR: -mg_levels_pc_type jacobi
[49]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[49]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[49]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[49]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[49]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[49]PETSC ERROR: -pc_gamg_process_eq_limit 400
[49]PETSC ERROR: -pc_gamg_repartition false
[49]PETSC ERROR: -pc_gamg_reuse_interpolation true
[49]PETSC ERROR: -pc_gamg_threshold 0.01
[49]PETSC ERROR: -pc_type gamg
[49]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[49]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[49]PETSC ERROR: -petscpartitioner_type simple
[49]PETSC ERROR: -potential_petscspace_degree 2
[49]PETSC ERROR: -snes_lag_jacobian -2
[42]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[42]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[42]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[42]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[42]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[42]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[42]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[42]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[42]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[50]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[50]PETSC ERROR: No support for this operation for this object type
[50]PETSC ERROR: No method productsymbolic for Mat of type (null)
[50]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[50]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[50]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[50]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[50]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[50]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[50]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[42]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[42]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[42]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[42]PETSC ERROR: #14 main() at ex13.c:178
[42]PETSC ERROR: PETSc Option Table entries:
[42]PETSC ERROR: -benchmark_it 10
[42]PETSC ERROR: -dm_distribute
[42]PETSC ERROR: -dm_mat_type aijkokkos
[42]PETSC ERROR: -dm_plex_box_faces 4,4,4
[42]PETSC ERROR: -dm_plex_box_lower 0,0,0
[42]PETSC ERROR: -dm_plex_box_upper 2,2,2
[42]PETSC ERROR: -dm_plex_dim 3
[42]PETSC ERROR: -dm_plex_simplex 0
[42]PETSC ERROR: -dm_refine 4
[42]PETSC ERROR: -dm_vec_type kokkos
[50]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[50]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[50]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[51]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[51]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[51]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[51]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[51]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[51]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[51]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[51]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[51]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[51]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[51]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[51]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[51]PETSC ERROR: #14 main() at ex13.c:178
[51]PETSC ERROR: PETSc Option Table entries:
[51]PETSC ERROR: -benchmark_it 10
[51]PETSC ERROR: -dm_distribute
[51]PETSC ERROR: -dm_mat_type aijkokkos
[51]PETSC ERROR: -dm_plex_box_faces 4,4,4
[51]PETSC ERROR: -dm_plex_box_lower 0,0,0
[51]PETSC ERROR: -dm_plex_box_upper 2,2,2
[51]PETSC ERROR: -dm_plex_dim 3
[51]PETSC ERROR: -dm_plex_simplex 0
[51]PETSC ERROR: -dm_refine 4
[51]PETSC ERROR: -dm_vec_type kokkos
[51]PETSC ERROR: -dm_view
[51]PETSC ERROR: -ksp_converged_reason
[51]PETSC ERROR: -ksp_max_it 200
[51]PETSC ERROR: -ksp_norm_type unpreconditioned
[51]PETSC ERROR: -ksp_rtol 1.e-12
[51]PETSC ERROR: -ksp_type cg
[51]PETSC ERROR: -log_view
[51]PETSC ERROR: -mat_type aijkokkos
[51]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[51]PETSC ERROR: -mg_levels_ksp_type chebyshev
[51]PETSC ERROR: -mg_levels_pc_type jacobi
[51]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[51]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[51]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[51]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[51]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[51]PETSC ERROR: -pc_gamg_process_eq_limit 400
[51]PETSC ERROR: -pc_gamg_repartition false
[51]PETSC ERROR: -pc_gamg_reuse_interpolation true
[51]PETSC ERROR: -pc_gamg_threshold 0.01
[51]PETSC ERROR: -pc_type gamg
[51]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[51]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[51]PETSC ERROR: -petscpartitioner_type simple
[51]PETSC ERROR: -potential_petscspace_degree 2
[51]PETSC ERROR: -snes_lag_jacobian -2
[51]PETSC ERROR: -snes_max_it 1
[51]PETSC ERROR: -snes_rtol 1.e-8
[51]PETSC ERROR: -snes_type ksponly
[51]PETSC ERROR: -use_gpu_aware_mpi 0
[51]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[22]PETSC ERROR: -dm_plex_box_upper 2,2,2
[22]PETSC ERROR: -dm_plex_dim 3
[22]PETSC ERROR: -dm_plex_simplex 0
[22]PETSC ERROR: -dm_refine 4
[22]PETSC ERROR: -dm_vec_type kokkos
[22]PETSC ERROR: -dm_view
[22]PETSC ERROR: -ksp_converged_reason
[22]PETSC ERROR: -ksp_max_it 200
[22]PETSC ERROR: -ksp_norm_type unpreconditioned
[22]PETSC ERROR: -ksp_rtol 1.e-12
[22]PETSC ERROR: -ksp_type cg
[22]PETSC ERROR: -log_view
[22]PETSC ERROR: -mat_type aijkokkos
[22]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[22]PETSC ERROR: -mg_levels_ksp_type chebyshev
[22]PETSC ERROR: -mg_levels_pc_type jacobi
[22]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[22]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[22]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[22]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[22]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[22]PETSC ERROR: -pc_gamg_process_eq_limit 400
[22]PETSC ERROR: -pc_gamg_repartition false
[22]PETSC ERROR: -pc_gamg_reuse_interpolation true
[22]PETSC ERROR: -pc_gamg_threshold 0.01
[22]PETSC ERROR: -pc_type gamg
[22]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[22]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[22]PETSC ERROR: -petscpartitioner_type simple
[22]PETSC ERROR: -potential_petscspace_degree 2
[22]PETSC ERROR: -snes_lag_jacobian -2
[22]PETSC ERROR: -snes_max_it 1
[22]PETSC ERROR: -snes_rtol 1.e-8
[22]PETSC ERROR: -snes_type ksponly
[22]PETSC ERROR: -use_gpu_aware_mpi 0
[22]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 22] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[23]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[23]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[23]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[23]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[23]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[23]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[23]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[23]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[23]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[23]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[23]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[23]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[12]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[12]PETSC ERROR: -mg_levels_ksp_type chebyshev
[12]PETSC ERROR: -mg_levels_pc_type jacobi
[12]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[12]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[12]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[12]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[12]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[12]PETSC ERROR: -pc_gamg_process_eq_limit 400
[12]PETSC ERROR: -pc_gamg_repartition false
[12]PETSC ERROR: -pc_gamg_reuse_interpolation true
[12]PETSC ERROR: -pc_gamg_threshold 0.01
[12]PETSC ERROR: -pc_type gamg
[12]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[12]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[12]PETSC ERROR: -petscpartitioner_type simple
[12]PETSC ERROR: -potential_petscspace_degree 2
[12]PETSC ERROR: -snes_lag_jacobian -2
[12]PETSC ERROR: -snes_max_it 1
[12]PETSC ERROR: -snes_rtol 1.e-8
[12]PETSC ERROR: -snes_type ksponly
[12]PETSC ERROR: -use_gpu_aware_mpi 0
[12]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 12] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[15]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[15]PETSC ERROR: -pc_gamg_process_eq_limit 400
[15]PETSC ERROR: -pc_gamg_repartition false
[15]PETSC ERROR: -pc_gamg_reuse_interpolation true
[15]PETSC ERROR: -pc_gamg_threshold 0.01
[15]PETSC ERROR: -pc_type gamg
[15]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[15]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[15]PETSC ERROR: -petscpartitioner_type simple
[15]PETSC ERROR: -potential_petscspace_degree 2
[15]PETSC ERROR: -snes_lag_jacobian -2
[15]PETSC ERROR: -snes_max_it 1
[15]PETSC ERROR: -snes_rtol 1.e-8
[15]PETSC ERROR: -snes_type ksponly
[15]PETSC ERROR: -use_gpu_aware_mpi 0
[15]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 15] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001117] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[35]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[35]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[35]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[35]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[35]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[35]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[35]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[35]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[35]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[35]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[35]PETSC ERROR: #14 main() at ex13.c:178
[35]PETSC ERROR: PETSc Option Table entries:
[35]PETSC ERROR: -benchmark_it 10
[35]PETSC ERROR: -dm_distribute
[35]PETSC ERROR: -dm_mat_type aijkokkos
[35]PETSC ERROR: -dm_plex_box_faces 4,4,4
[35]PETSC ERROR: -dm_plex_box_lower 0,0,0
[35]PETSC ERROR: -dm_plex_box_upper 2,2,2
[35]PETSC ERROR: -dm_plex_dim 3
[35]PETSC ERROR: -dm_plex_simplex 0
[35]PETSC ERROR: -dm_refine 4
[35]PETSC ERROR: -dm_vec_type kokkos
[35]PETSC ERROR: -dm_view
[35]PETSC ERROR: -ksp_converged_reason
[35]PETSC ERROR: -ksp_max_it 200
[35]PETSC ERROR: -ksp_norm_type unpreconditioned
[35]PETSC ERROR: -ksp_rtol 1.e-12
[35]PETSC ERROR: -ksp_type cg
[35]PETSC ERROR: -log_view
[35]PETSC ERROR: -mat_type aijkokkos
[35]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[35]PETSC ERROR: -mg_levels_ksp_type chebyshev
[35]PETSC ERROR: -mg_levels_pc_type jacobi
[35]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[35]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[35]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[35]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[36]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[36]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[36]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[36]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[36]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[36]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[36]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[36]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[36]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[26]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[26]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[26]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[26]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[26]PETSC ERROR: #14 main() at ex13.c:178
[26]PETSC ERROR: PETSc Option Table entries:
[26]PETSC ERROR: -benchmark_it 10
[26]PETSC ERROR: -dm_distribute
[26]PETSC ERROR: -dm_mat_type aijkokkos
[26]PETSC ERROR: -dm_plex_box_faces 4,4,4
[26]PETSC ERROR: -dm_plex_box_lower 0,0,0
[26]PETSC ERROR: -dm_plex_box_upper 2,2,2
[26]PETSC ERROR: -dm_plex_dim 3
[26]PETSC ERROR: -dm_plex_simplex 0
[26]PETSC ERROR: -dm_refine 4
[26]PETSC ERROR: -dm_vec_type kokkos
[26]PETSC ERROR: -dm_view
[26]PETSC ERROR: -ksp_converged_reason
[26]PETSC ERROR: -ksp_max_it 200
[26]PETSC ERROR: -ksp_norm_type unpreconditioned
[43]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[43]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[43]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[43]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[43]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[43]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[43]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[43]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[43]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[26]PETSC ERROR: -ksp_rtol 1.e-12
[26]PETSC ERROR: -ksp_type cg
[26]PETSC ERROR: -log_view
[26]PETSC ERROR: -mat_type aijkokkos
[26]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[26]PETSC ERROR: -mg_levels_ksp_type chebyshev
[26]PETSC ERROR: -mg_levels_pc_type jacobi
[26]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[26]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[26]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[26]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[26]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[26]PETSC ERROR: -pc_gamg_process_eq_limit 400
[26]PETSC ERROR: -pc_gamg_repartition false
[26]PETSC ERROR: -pc_gamg_reuse_interpolation true
[26]PETSC ERROR: -pc_gamg_threshold 0.01
[26]PETSC ERROR: -pc_type gamg
[26]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[26]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[26]PETSC ERROR: -petscpartitioner_type simple
[26]PETSC ERROR: -potential_petscspace_degree 2
[26]PETSC ERROR: -snes_lag_jacobian -2
[43]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[43]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[43]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[43]PETSC ERROR: #14 main() at ex13.c:178
[43]PETSC ERROR: PETSc Option Table entries:
[43]PETSC ERROR: -benchmark_it 10
[43]PETSC ERROR: -dm_distribute
[43]PETSC ERROR: -dm_mat_type aijkokkos
[43]PETSC ERROR: -dm_plex_box_faces 4,4,4
[43]PETSC ERROR: -dm_plex_box_lower 0,0,0
[43]PETSC ERROR: -dm_plex_box_upper 2,2,2
[43]PETSC ERROR: -dm_plex_dim 3
[43]PETSC ERROR: -dm_plex_simplex 0
[43]PETSC ERROR: -dm_refine 4
[43]PETSC ERROR: -dm_vec_type kokkos
[43]PETSC ERROR: -dm_view
[43]PETSC ERROR: -ksp_converged_reason
[43]PETSC ERROR: -ksp_max_it 200
[43]PETSC ERROR: -ksp_norm_type unpreconditioned
[43]PETSC ERROR: -ksp_rtol 1.e-12
[43]PETSC ERROR: -ksp_type cg
[43]PETSC ERROR: -log_view
[26]PETSC ERROR: -snes_max_it 1
[26]PETSC ERROR: -snes_rtol 1.e-8
[26]PETSC ERROR: -snes_type ksponly
[26]PETSC ERROR: -use_gpu_aware_mpi 0
[26]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 26] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[43]PETSC ERROR: -mat_type aijkokkos
[43]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[43]PETSC ERROR: -mg_levels_ksp_type chebyshev
[43]PETSC ERROR: -mg_levels_pc_type jacobi
[43]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[43]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[43]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[43]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[43]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[43]PETSC ERROR: -pc_gamg_process_eq_limit 400
[43]PETSC ERROR: -pc_gamg_repartition false
[43]PETSC ERROR: -pc_gamg_reuse_interpolation true
[43]PETSC ERROR: -pc_gamg_threshold 0.01
[43]PETSC ERROR: -pc_type gamg
[43]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[43]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[43]PETSC ERROR: -petscpartitioner_type simple
[43]PETSC ERROR: -potential_petscspace_degree 2
[43]PETSC ERROR: -snes_lag_jacobian -2
[43]PETSC ERROR: -snes_max_it 1
[43]PETSC ERROR: -snes_rtol 1.e-8
[43]PETSC ERROR: -snes_type ksponly
[27]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[27]PETSC ERROR: No support for this operation for this object type
[27]PETSC ERROR: No method productsymbolic for Mat of type (null)
[27]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[27]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[27]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[27]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[27]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[27]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[27]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[43]PETSC ERROR: -use_gpu_aware_mpi 0
[43]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 43] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[59]PETSC ERROR: -use_gpu_aware_mpi 0
[61]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[61]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[61]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[61]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[61]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[61]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[61]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[61]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[61]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[61]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[61]PETSC ERROR: #14 main() at ex13.c:178
[61]PETSC ERROR: PETSc Option Table entries:
[61]PETSC ERROR: -benchmark_it 10
[61]PETSC ERROR: -dm_distribute
[61]PETSC ERROR: -dm_mat_type aijkokkos
[61]PETSC ERROR: -dm_plex_box_faces 4,4,4
[61]PETSC ERROR: -dm_plex_box_lower 0,0,0
[61]PETSC ERROR: -dm_plex_box_upper 2,2,2
[61]PETSC ERROR: -dm_plex_dim 3
[61]PETSC ERROR: -dm_plex_simplex 0
[61]PETSC ERROR: -dm_refine 4
[61]PETSC ERROR: -dm_vec_type kokkos
[61]PETSC ERROR: -dm_view
[61]PETSC ERROR: -ksp_converged_reason
[61]PETSC ERROR: -ksp_max_it 200
[61]PETSC ERROR: -ksp_norm_type unpreconditioned
[61]PETSC ERROR: -ksp_rtol 1.e-12
[61]PETSC ERROR: -ksp_type cg
[61]PETSC ERROR: -log_view
[61]PETSC ERROR: -mat_type aijkokkos
[61]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[61]PETSC ERROR: -mg_levels_ksp_type chebyshev
[61]PETSC ERROR: -mg_levels_pc_type jacobi
[61]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[61]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[61]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[61]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[61]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[61]PETSC ERROR: -pc_gamg_process_eq_limit 400
[61]PETSC ERROR: -pc_gamg_repartition false
[61]PETSC ERROR: -pc_gamg_reuse_interpolation true
[61]PETSC ERROR: -pc_gamg_threshold 0.01
[61]PETSC ERROR: -pc_type gamg
[61]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[61]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[61]PETSC ERROR: -petscpartitioner_type simple
[61]PETSC ERROR: -potential_petscspace_degree 2
[61]PETSC ERROR: -snes_lag_jacobian -2
[61]PETSC ERROR: -snes_max_it 1
[61]PETSC ERROR: -snes_rtol 1.e-8
[61]PETSC ERROR: -snes_type ksponly
[61]PETSC ERROR: -use_gpu_aware_mpi 0
[61]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[63]PETSC ERROR: -ksp_max_it 200
[63]PETSC ERROR: -ksp_norm_type unpreconditioned
[63]PETSC ERROR: -ksp_rtol 1.e-12
[63]PETSC ERROR: -ksp_type cg
[63]PETSC ERROR: -log_view
[63]PETSC ERROR: -mat_type aijkokkos
[63]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[63]PETSC ERROR: -mg_levels_ksp_type chebyshev
[63]PETSC ERROR: -mg_levels_pc_type jacobi
[63]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[63]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[63]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[63]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[63]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[63]PETSC ERROR: -pc_gamg_process_eq_limit 400
[63]PETSC ERROR: -pc_gamg_repartition false
[63]PETSC ERROR: -pc_gamg_reuse_interpolation true
[63]PETSC ERROR: -pc_gamg_threshold 0.01
[63]PETSC ERROR: -pc_type gamg
[63]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[63]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[63]PETSC ERROR: -petscpartitioner_type simple
[36]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[36]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[36]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[36]PETSC ERROR: #14 main() at ex13.c:178
[36]PETSC ERROR: PETSc Option Table entries:
[36]PETSC ERROR: -benchmark_it 10
[36]PETSC ERROR: -dm_distribute
[36]PETSC ERROR: -dm_mat_type aijkokkos
[36]PETSC ERROR: -dm_plex_box_faces 4,4,4
[36]PETSC ERROR: -dm_plex_box_lower 0,0,0
[36]PETSC ERROR: -dm_plex_box_upper 2,2,2
[36]PETSC ERROR: -dm_plex_dim 3
[36]PETSC ERROR: -dm_plex_simplex 0
[36]PETSC ERROR: -dm_refine 4
[36]PETSC ERROR: -dm_vec_type kokkos
[36]PETSC ERROR: -dm_view
[36]PETSC ERROR: -ksp_converged_reason
[36]PETSC ERROR: -ksp_max_it 200
[36]PETSC ERROR: -ksp_norm_type unpreconditioned
[36]PETSC ERROR: -ksp_rtol 1.e-12
[36]PETSC ERROR: -ksp_type cg
[36]PETSC ERROR: -log_view
[63]PETSC ERROR: -potential_petscspace_degree 2
[63]PETSC ERROR: -snes_lag_jacobian -2
[63]PETSC ERROR: -snes_max_it 1
[63]PETSC ERROR: -snes_rtol 1.e-8
[63]PETSC ERROR: -snes_type ksponly
[63]PETSC ERROR: -use_gpu_aware_mpi 0
[63]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 63] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[36]PETSC ERROR: -mat_type aijkokkos
[36]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[36]PETSC ERROR: -mg_levels_ksp_type chebyshev
[36]PETSC ERROR: -mg_levels_pc_type jacobi
[36]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[36]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[36]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[36]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[36]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[36]PETSC ERROR: -pc_gamg_process_eq_limit 400
[36]PETSC ERROR: -pc_gamg_repartition false
[36]PETSC ERROR: -pc_gamg_reuse_interpolation true
[36]PETSC ERROR: -pc_gamg_threshold 0.01
[36]PETSC ERROR: -pc_type gamg
[36]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[36]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[36]PETSC ERROR: -petscpartitioner_type simple
[36]PETSC ERROR: -potential_petscspace_degree 2
[36]PETSC ERROR: -snes_lag_jacobian -2
[36]PETSC ERROR: -snes_max_it 1
[36]PETSC ERROR: -snes_rtol 1.e-8
[56]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[56]PETSC ERROR: No support for this operation for this object type
[56]PETSC ERROR: No method productsymbolic for Mat of type (null)
[56]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[56]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[56]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[56]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[56]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[56]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[56]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
MPICH ERROR [Rank 38] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[56]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[56]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[56]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[39]PETSC ERROR: -dm_mat_type aijkokkos
[39]PETSC ERROR: -dm_plex_box_faces 4,4,4
[39]PETSC ERROR: -dm_plex_box_lower 0,0,0
[39]PETSC ERROR: -dm_plex_box_upper 2,2,2
[39]PETSC ERROR: -dm_plex_dim 3
[39]PETSC ERROR: -dm_plex_simplex 0
[39]PETSC ERROR: -dm_refine 4
[39]PETSC ERROR: -dm_vec_type kokkos
[39]PETSC ERROR: -dm_view
[39]PETSC ERROR: -ksp_converged_reason
[39]PETSC ERROR: -ksp_max_it 200
[39]PETSC ERROR: -ksp_norm_type unpreconditioned
[39]PETSC ERROR: -ksp_rtol 1.e-12
[39]PETSC ERROR: -ksp_type cg
[39]PETSC ERROR: -log_view
[39]PETSC ERROR: -mat_type aijkokkos
[39]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[39]PETSC ERROR: -mg_levels_ksp_type chebyshev
[39]PETSC ERROR: -mg_levels_pc_type jacobi
[39]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[39]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[39]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[39]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[39]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[39]PETSC ERROR: -pc_gamg_process_eq_limit 400
[56]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[56]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[56]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[56]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[56]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[56]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[56]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[39]PETSC ERROR: -pc_gamg_repartition false
[39]PETSC ERROR: -pc_gamg_reuse_interpolation true
[39]PETSC ERROR: -pc_gamg_threshold 0.01
[39]PETSC ERROR: -pc_type gamg
[39]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[39]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[39]PETSC ERROR: -petscpartitioner_type simple
[39]PETSC ERROR: -potential_petscspace_degree 2
[39]PETSC ERROR: -snes_lag_jacobian -2
[39]PETSC ERROR: -snes_max_it 1
[39]PETSC ERROR: -snes_rtol 1.e-8
[39]PETSC ERROR: -snes_type ksponly
[39]PETSC ERROR: -use_gpu_aware_mpi 0
[39]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 39] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: No support for this operation for this object type
[32]PETSC ERROR: No method productsymbolic for Mat of type (null)
[32]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[32]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[32]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[32]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[32]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[32]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[32]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[32]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[32]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[32]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[32]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[32]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[32]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[32]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[32]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[33]PETSC ERROR: -ksp_norm_type unpreconditioned
[33]PETSC ERROR: -ksp_rtol 1.e-12
[33]PETSC ERROR: -ksp_type cg
[33]PETSC ERROR: -log_view
[33]PETSC ERROR: -mat_type aijkokkos
[33]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[33]PETSC ERROR: -mg_levels_ksp_type chebyshev
[33]PETSC ERROR: -mg_levels_pc_type jacobi
[33]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[33]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[33]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[33]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[33]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[33]PETSC ERROR: -pc_gamg_process_eq_limit 400
[33]PETSC ERROR: -pc_gamg_repartition false
[33]PETSC ERROR: -pc_gamg_reuse_interpolation true
[33]PETSC ERROR: -pc_gamg_threshold 0.01
[33]PETSC ERROR: -pc_type gamg
[33]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[33]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[33]PETSC ERROR: -petscpartitioner_type simple
[33]PETSC ERROR: -potential_petscspace_degree 2
[27]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[27]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[27]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[27]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[27]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[27]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[27]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[27]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[27]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[27]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[27]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[27]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[27]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[27]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[27]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[27]PETSC ERROR: #14 main() at ex13.c:178
[27]PETSC ERROR: PETSc Option Table entries:
[27]PETSC ERROR: -benchmark_it 10
[27]PETSC ERROR: -dm_distribute
[27]PETSC ERROR: -dm_mat_type aijkokkos
[27]PETSC ERROR: -dm_plex_box_faces 4,4,4
[27]PETSC ERROR: -dm_plex_box_lower 0,0,0
[27]PETSC ERROR: -dm_plex_box_upper 2,2,2
[27]PETSC ERROR: -dm_plex_dim 3
[27]PETSC ERROR: -dm_plex_simplex 0
[27]PETSC ERROR: -dm_refine 4
[27]PETSC ERROR: -dm_vec_type kokkos
[27]PETSC ERROR: -dm_view
[27]PETSC ERROR: -ksp_converged_reason
[27]PETSC ERROR: -ksp_max_it 200
[27]PETSC ERROR: -ksp_norm_type unpreconditioned
[27]PETSC ERROR: -ksp_rtol 1.e-12
[27]PETSC ERROR: -ksp_type cg
[27]PETSC ERROR: -log_view
[23]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[23]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[23]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[23]PETSC ERROR: #14 main() at ex13.c:178
[23]PETSC ERROR: PETSc Option Table entries:
[23]PETSC ERROR: -benchmark_it 10
[23]PETSC ERROR: -dm_distribute
[23]PETSC ERROR: -dm_mat_type aijkokkos
[23]PETSC ERROR: -dm_plex_box_faces 4,4,4
[23]PETSC ERROR: -dm_plex_box_lower 0,0,0
[23]PETSC ERROR: -dm_plex_box_upper 2,2,2
[23]PETSC ERROR: -dm_plex_dim 3
[23]PETSC ERROR: -dm_plex_simplex 0
[23]PETSC ERROR: -dm_refine 4
[23]PETSC ERROR: -dm_vec_type kokkos
[23]PETSC ERROR: -dm_view
[23]PETSC ERROR: -ksp_converged_reason
[23]PETSC ERROR: -ksp_max_it 200
[23]PETSC ERROR: -ksp_norm_type unpreconditioned
[23]PETSC ERROR: -ksp_rtol 1.e-12
[23]PETSC ERROR: -ksp_type cg
[23]PETSC ERROR: -log_view
[23]PETSC ERROR: -mat_type aijkokkos
[23]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[23]PETSC ERROR: -mg_levels_ksp_type chebyshev
[23]PETSC ERROR: -mg_levels_pc_type jacobi
[23]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[23]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[23]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[23]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[23]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[23]PETSC ERROR: -pc_gamg_process_eq_limit 400
[23]PETSC ERROR: -pc_gamg_repartition false
[23]PETSC ERROR: -pc_gamg_reuse_interpolation true
[23]PETSC ERROR: -pc_gamg_threshold 0.01
[23]PETSC ERROR: -pc_type gamg
[23]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[23]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[23]PETSC ERROR: -petscpartitioner_type simple
[23]PETSC ERROR: -potential_petscspace_degree 2
[23]PETSC ERROR: -snes_lag_jacobian -2
[23]PETSC ERROR: -snes_max_it 1
[23]PETSC ERROR: -snes_rtol 1.e-8
[23]PETSC ERROR: -snes_type ksponly
[23]PETSC ERROR: -use_gpu_aware_mpi 0
[23]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: No support for this operation for this object type
[16]PETSC ERROR: No method productsymbolic for Mat of type (null)
[16]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[16]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[16]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[16]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[16]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[16]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[16]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[57]PETSC ERROR: -mat_type aijkokkos
[57]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[57]PETSC ERROR: -mg_levels_ksp_type chebyshev
[57]PETSC ERROR: -mg_levels_pc_type jacobi
[57]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[57]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[57]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[57]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[57]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[57]PETSC ERROR: -pc_gamg_process_eq_limit 400
[57]PETSC ERROR: -pc_gamg_repartition false
[57]PETSC ERROR: -pc_gamg_reuse_interpolation true
[57]PETSC ERROR: -pc_gamg_threshold 0.01
[57]PETSC ERROR: -pc_type gamg
[57]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[57]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[57]PETSC ERROR: -petscpartitioner_type simple
[57]PETSC ERROR: -potential_petscspace_degree 2
[57]PETSC ERROR: -snes_lag_jacobian -2
[57]PETSC ERROR: -snes_max_it 1
[57]PETSC ERROR: -snes_rtol 1.e-8
[57]PETSC ERROR: -snes_type ksponly
[57]PETSC ERROR: -use_gpu_aware_mpi 0
[57]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 57] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[27]PETSC ERROR: -mat_type aijkokkos
[27]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[27]PETSC ERROR: -mg_levels_ksp_type chebyshev
[27]PETSC ERROR: -mg_levels_pc_type jacobi
[27]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[27]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[27]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[27]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
MPICH ERROR [Rank 52] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[28]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[28]PETSC ERROR: No support for this operation for this object type
[28]PETSC ERROR: No method productsymbolic for Mat of type (null)
[28]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[28]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[28]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[28]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[28]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[28]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[28]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[53]PETSC ERROR: -ksp_converged_reason
[53]PETSC ERROR: -ksp_max_it 200
[53]PETSC ERROR: -ksp_norm_type unpreconditioned
[53]PETSC ERROR: -ksp_rtol 1.e-12
[53]PETSC ERROR: -ksp_type cg
[53]PETSC ERROR: -log_view
[53]PETSC ERROR: -mat_type aijkokkos
[53]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[53]PETSC ERROR: -mg_levels_ksp_type chebyshev
[53]PETSC ERROR: -mg_levels_pc_type jacobi
[53]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[53]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[53]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[53]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[53]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[53]PETSC ERROR: -pc_gamg_process_eq_limit 400
[53]PETSC ERROR: -pc_gamg_repartition false
[53]PETSC ERROR: -pc_gamg_reuse_interpolation true
[53]PETSC ERROR: -pc_gamg_threshold 0.01
[53]PETSC ERROR: -pc_type gamg
[53]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[53]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[28]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[28]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[28]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[53]PETSC ERROR: -petscpartitioner_type simple
[53]PETSC ERROR: -potential_petscspace_degree 2
[53]PETSC ERROR: -snes_lag_jacobian -2
[53]PETSC ERROR: -snes_max_it 1
[53]PETSC ERROR: -snes_rtol 1.e-8
[53]PETSC ERROR: -snes_type ksponly
[53]PETSC ERROR: -use_gpu_aware_mpi 0
[53]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 53] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[28]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[28]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[28]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[28]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[28]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[28]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[28]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[28]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[28]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[55]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[55]PETSC ERROR: -pc_gamg_process_eq_limit 400
[55]PETSC ERROR: -pc_gamg_repartition false
[55]PETSC ERROR: -pc_gamg_reuse_interpolation true
[55]PETSC ERROR: -pc_gamg_threshold 0.01
[55]PETSC ERROR: -pc_type gamg
[55]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[55]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[55]PETSC ERROR: -petscpartitioner_type simple
[55]PETSC ERROR: -potential_petscspace_degree 2
[55]PETSC ERROR: -snes_lag_jacobian -2
[55]PETSC ERROR: -snes_max_it 1
[55]PETSC ERROR: -snes_rtol 1.e-8
[55]PETSC ERROR: -snes_type ksponly
[55]PETSC ERROR: -use_gpu_aware_mpi 0
[55]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 55] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[48]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[48]PETSC ERROR: No support for this operation for this object type
[48]PETSC ERROR: No method productsymbolic for Mat of type (null)
[48]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[48]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[48]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[48]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[48]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[48]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[48]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[48]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001128 by madams Sun Oct 30 05:30:05 2022
[48]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[48]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[48]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[48]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[48]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[48]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[48]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[48]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[48]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[48]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[48]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[48]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[48]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[48]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[48]PETSC ERROR: #14 main() at ex13.c:178
[48]PETSC ERROR: PETSc Option Table entries:
[48]PETSC ERROR: -benchmark_it 10
[49]PETSC ERROR: -snes_max_it 1
[49]PETSC ERROR: -snes_rtol 1.e-8
[49]PETSC ERROR: -snes_type ksponly
[49]PETSC ERROR: -use_gpu_aware_mpi 0
[49]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 49] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[50]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[50]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[50]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[50]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[50]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[50]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[50]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[50]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[50]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[50]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[50]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[50]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[50]PETSC ERROR: #14 main() at ex13.c:178
[50]PETSC ERROR: PETSc Option Table entries:
[50]PETSC ERROR: -benchmark_it 10
[50]PETSC ERROR: -dm_distribute
[50]PETSC ERROR: -dm_mat_type aijkokkos
[50]PETSC ERROR: -dm_plex_box_faces 4,4,4
[50]PETSC ERROR: -dm_plex_box_lower 0,0,0
[50]PETSC ERROR: -dm_plex_box_upper 2,2,2
[50]PETSC ERROR: -dm_plex_dim 3
[50]PETSC ERROR: -dm_plex_simplex 0
[50]PETSC ERROR: -dm_refine 4
[50]PETSC ERROR: -dm_vec_type kokkos
[50]PETSC ERROR: -dm_view
[50]PETSC ERROR: -ksp_converged_reason
[50]PETSC ERROR: -ksp_max_it 200
[50]PETSC ERROR: -ksp_norm_type unpreconditioned
[50]PETSC ERROR: -ksp_rtol 1.e-12
[50]PETSC ERROR: -ksp_type cg
[50]PETSC ERROR: -log_view
[50]PETSC ERROR: -mat_type aijkokkos
[50]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[50]PETSC ERROR: -mg_levels_ksp_type chebyshev
[50]PETSC ERROR: -mg_levels_pc_type jacobi
[50]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[50]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[50]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
MPICH ERROR [Rank 51] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[48]PETSC ERROR: -dm_distribute
[48]PETSC ERROR: -dm_mat_type aijkokkos
[48]PETSC ERROR: -dm_plex_box_faces 4,4,4
[48]PETSC ERROR: -dm_plex_box_lower 0,0,0
[48]PETSC ERROR: -dm_plex_box_upper 2,2,2
[48]PETSC ERROR: -dm_plex_dim 3
[48]PETSC ERROR: -dm_plex_simplex 0
[48]PETSC ERROR: -dm_refine 4
[48]PETSC ERROR: -dm_vec_type kokkos
[48]PETSC ERROR: -dm_view
[48]PETSC ERROR: -ksp_converged_reason
[48]PETSC ERROR: -ksp_max_it 200
[48]PETSC ERROR: -ksp_norm_type unpreconditioned
[48]PETSC ERROR: -ksp_rtol 1.e-12
[48]PETSC ERROR: -ksp_type cg
[48]PETSC ERROR: -log_view
[48]PETSC ERROR: -mat_type aijkokkos
[48]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[48]PETSC ERROR: -mg_levels_ksp_type chebyshev
[48]PETSC ERROR: -mg_levels_pc_type jacobi
[48]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[48]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[48]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[48]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[48]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[16]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[16]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[16]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[17]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[17]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[17]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[17]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[17]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[17]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[17]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[17]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[17]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[17]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[33]PETSC ERROR: -snes_lag_jacobian -2
[33]PETSC ERROR: -snes_max_it 1
[33]PETSC ERROR: -snes_rtol 1.e-8
[33]PETSC ERROR: -snes_type ksponly
[33]PETSC ERROR: -use_gpu_aware_mpi 0
[33]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 33] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[34]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[34]PETSC ERROR: No support for this operation for this object type
[34]PETSC ERROR: No method productsymbolic for Mat of type (null)
[34]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[34]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[34]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[34]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[34]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[34]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[34]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[34]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[34]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[34]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[34]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[34]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[34]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[34]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[34]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[34]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[34]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[34]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[34]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[34]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[34]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[34]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[34]PETSC ERROR: #14 main() at ex13.c:178
[34]PETSC ERROR: PETSc Option Table entries:
[34]PETSC ERROR: -benchmark_it 10
[34]PETSC ERROR: -dm_distribute
[34]PETSC ERROR: -dm_mat_type aijkokkos
[34]PETSC ERROR: -dm_plex_box_faces 4,4,4
[34]PETSC ERROR: -dm_plex_box_lower 0,0,0
[34]PETSC ERROR: -dm_plex_box_upper 2,2,2
[34]PETSC ERROR: -dm_plex_dim 3
[34]PETSC ERROR: -dm_plex_simplex 0
[34]PETSC ERROR: -dm_refine 4
[34]PETSC ERROR: -dm_vec_type kokkos
[34]PETSC ERROR: -dm_view
[34]PETSC ERROR: -ksp_converged_reason
[34]PETSC ERROR: -ksp_max_it 200
[48]PETSC ERROR: -pc_gamg_process_eq_limit 400
[48]PETSC ERROR: -pc_gamg_repartition false
[48]PETSC ERROR: -pc_gamg_reuse_interpolation true
[48]PETSC ERROR: -pc_gamg_threshold 0.01
[48]PETSC ERROR: -pc_type gamg
[48]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[48]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[48]PETSC ERROR: -petscpartitioner_type simple
[48]PETSC ERROR: -potential_petscspace_degree 2
[44]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[44]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[44]PETSC ERROR: #14 main() at ex13.c:178
[44]PETSC ERROR: PETSc Option Table entries:
[44]PETSC ERROR: -benchmark_it 10
[44]PETSC ERROR: -dm_distribute
[44]PETSC ERROR: -dm_mat_type aijkokkos
[44]PETSC ERROR: -dm_plex_box_faces 4,4,4
[44]PETSC ERROR: -dm_plex_box_lower 0,0,0
[44]PETSC ERROR: -dm_plex_box_upper 2,2,2
[44]PETSC ERROR: -dm_plex_dim 3
[44]PETSC ERROR: -dm_plex_simplex 0
[44]PETSC ERROR: -dm_refine 4
[44]PETSC ERROR: -dm_vec_type kokkos
[44]PETSC ERROR: -dm_view
[44]PETSC ERROR: -ksp_converged_reason
[44]PETSC ERROR: -ksp_max_it 200
[44]PETSC ERROR: -ksp_norm_type unpreconditioned
[44]PETSC ERROR: -ksp_rtol 1.e-12
[44]PETSC ERROR: -ksp_type cg
[44]PETSC ERROR: -log_view
[44]PETSC ERROR: -mat_type aijkokkos
[44]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[44]PETSC ERROR: -mg_levels_ksp_type chebyshev
[44]PETSC ERROR: -mg_levels_pc_type jacobi
[44]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[44]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[44]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[44]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[44]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[44]PETSC ERROR: -pc_gamg_process_eq_limit 400
[44]PETSC ERROR: -pc_gamg_repartition false
[44]PETSC ERROR: -pc_gamg_reuse_interpolation true
[44]PETSC ERROR: -pc_gamg_threshold 0.01
[44]PETSC ERROR: -pc_type gamg
[44]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[44]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[44]PETSC ERROR: -petscpartitioner_type simple
[44]PETSC ERROR: -potential_petscspace_degree 2
[44]PETSC ERROR: -snes_lag_jacobian -2
[44]PETSC ERROR: -snes_max_it 1
[44]PETSC ERROR: -snes_rtol 1.e-8
[44]PETSC ERROR: -snes_type ksponly
[44]PETSC ERROR: -use_gpu_aware_mpi 0
[50]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[50]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[50]PETSC ERROR: -pc_gamg_process_eq_limit 400
[50]PETSC ERROR: -pc_gamg_repartition false
[50]PETSC ERROR: -pc_gamg_reuse_interpolation true
[50]PETSC ERROR: -pc_gamg_threshold 0.01
[50]PETSC ERROR: -pc_type gamg
[50]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[50]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[50]PETSC ERROR: -petscpartitioner_type simple
[50]PETSC ERROR: -potential_petscspace_degree 2
[50]PETSC ERROR: -snes_lag_jacobian -2
[50]PETSC ERROR: -snes_max_it 1
[50]PETSC ERROR: -snes_rtol 1.e-8
[50]PETSC ERROR: -snes_type ksponly
[50]PETSC ERROR: -use_gpu_aware_mpi 0
[50]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 50] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[44]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 44] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[45]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[45]PETSC ERROR: No support for this operation for this object type
[45]PETSC ERROR: No method productsymbolic for Mat of type (null)
[45]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[45]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[45]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[45]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[45]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[45]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[45]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[48]PETSC ERROR: -snes_lag_jacobian -2
[48]PETSC ERROR: -snes_max_it 1
[48]PETSC ERROR: -snes_rtol 1.e-8
[48]PETSC ERROR: -snes_type ksponly
[48]PETSC ERROR: -use_gpu_aware_mpi 0
[48]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 48] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001128] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[58]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[58]PETSC ERROR: No support for this operation for this object type
[58]PETSC ERROR: No method productsymbolic for Mat of type (null)
[58]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[58]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[58]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[58]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[58]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[58]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[58]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[58]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[58]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[58]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[58]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[58]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[58]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[58]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[58]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[58]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[58]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[58]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[58]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[58]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[58]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[58]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[58]PETSC ERROR: #14 main() at ex13.c:178
[58]PETSC ERROR: PETSc Option Table entries:
[58]PETSC ERROR: -benchmark_it 10
[58]PETSC ERROR: -dm_distribute
[58]PETSC ERROR: -dm_mat_type aijkokkos
[58]PETSC ERROR: -dm_plex_box_faces 4,4,4
[58]PETSC ERROR: -dm_plex_box_lower 0,0,0
[58]PETSC ERROR: -dm_plex_box_upper 2,2,2
[58]PETSC ERROR: -dm_plex_dim 3
[58]PETSC ERROR: -dm_plex_simplex 0
[58]PETSC ERROR: -dm_refine 4
[58]PETSC ERROR: -dm_vec_type kokkos
[58]PETSC ERROR: -dm_view
[58]PETSC ERROR: -ksp_converged_reason
[58]PETSC ERROR: -ksp_max_it 200
[58]PETSC ERROR: -ksp_norm_type unpreconditioned
[58]PETSC ERROR: -ksp_rtol 1.e-12
[58]PETSC ERROR: -ksp_type cg
[58]PETSC ERROR: -log_view
[58]PETSC ERROR: -mat_type aijkokkos
[58]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[58]PETSC ERROR: -mg_levels_ksp_type chebyshev
[58]PETSC ERROR: -mg_levels_pc_type jacobi
[58]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[58]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[58]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[58]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[59]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 59] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[60]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[60]PETSC ERROR: No support for this operation for this object type
[60]PETSC ERROR: No method productsymbolic for Mat of type (null)
[60]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[60]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[60]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[60]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[60]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[60]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[60]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[60]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001129 by madams Sun Oct 30 05:30:05 2022
[60]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[60]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[60]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[60]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[60]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[60]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[60]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[60]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[60]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[60]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[60]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[60]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[60]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[60]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[60]PETSC ERROR: #14 main() at ex13.c:178
[60]PETSC ERROR: PETSc Option Table entries:
[60]PETSC ERROR: -benchmark_it 10
[60]PETSC ERROR: -dm_distribute
[60]PETSC ERROR: -dm_mat_type aijkokkos
[60]PETSC ERROR: -dm_plex_box_faces 4,4,4
[60]PETSC ERROR: -dm_plex_box_lower 0,0,0
[60]PETSC ERROR: -dm_plex_box_upper 2,2,2
[60]PETSC ERROR: -dm_plex_dim 3
[60]PETSC ERROR: -dm_plex_simplex 0
[60]PETSC ERROR: -dm_refine 4
[60]PETSC ERROR: -dm_vec_type kokkos
[60]PETSC ERROR: -dm_view
[60]PETSC ERROR: -ksp_converged_reason
[60]PETSC ERROR: -ksp_max_it 200
[60]PETSC ERROR: -ksp_norm_type unpreconditioned
[60]PETSC ERROR: -ksp_rtol 1.e-12
[60]PETSC ERROR: -ksp_type cg
[60]PETSC ERROR: -log_view
[60]PETSC ERROR: -mat_type aijkokkos
[60]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[60]PETSC ERROR: -mg_levels_ksp_type chebyshev
[60]PETSC ERROR: -mg_levels_pc_type jacobi
[60]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[60]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[60]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[60]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
MPICH ERROR [Rank 61] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[56]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[56]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[56]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[56]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[56]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[56]PETSC ERROR: #14 main() at ex13.c:178
[56]PETSC ERROR: PETSc Option Table entries:
[56]PETSC ERROR: -benchmark_it 10
[56]PETSC ERROR: -dm_distribute
[56]PETSC ERROR: -dm_mat_type aijkokkos
[56]PETSC ERROR: -dm_plex_box_faces 4,4,4
[56]PETSC ERROR: -dm_plex_box_lower 0,0,0
[56]PETSC ERROR: -dm_plex_box_upper 2,2,2
[56]PETSC ERROR: -dm_plex_dim 3
[56]PETSC ERROR: -dm_plex_simplex 0
[56]PETSC ERROR: -dm_refine 4
[56]PETSC ERROR: -dm_vec_type kokkos
[56]PETSC ERROR: -dm_view
[56]PETSC ERROR: -ksp_converged_reason
[56]PETSC ERROR: -ksp_max_it 200
[56]PETSC ERROR: -ksp_norm_type unpreconditioned
[56]PETSC ERROR: -ksp_rtol 1.e-12
[56]PETSC ERROR: -ksp_type cg
[56]PETSC ERROR: -log_view
[56]PETSC ERROR: -mat_type aijkokkos
[56]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[56]PETSC ERROR: -mg_levels_ksp_type chebyshev
[56]PETSC ERROR: -mg_levels_pc_type jacobi
[56]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[56]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[56]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[56]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[56]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[56]PETSC ERROR: -pc_gamg_process_eq_limit 400
[56]PETSC ERROR: -pc_gamg_repartition false
[56]PETSC ERROR: -pc_gamg_reuse_interpolation true
[56]PETSC ERROR: -pc_gamg_threshold 0.01
[56]PETSC ERROR: -pc_type gamg
[56]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[56]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[56]PETSC ERROR: -petscpartitioner_type simple
[56]PETSC ERROR: -potential_petscspace_degree 2
[56]PETSC ERROR: -snes_lag_jacobian -2
[56]PETSC ERROR: -snes_max_it 1
[56]PETSC ERROR: -snes_rtol 1.e-8
[56]PETSC ERROR: -snes_type ksponly
[56]PETSC ERROR: -use_gpu_aware_mpi 0
[56]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 56] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[17]PETSC ERROR: #14 main() at ex13.c:178
[17]PETSC ERROR: PETSc Option Table entries:
[17]PETSC ERROR: -benchmark_it 10
[17]PETSC ERROR: -dm_distribute
[17]PETSC ERROR: -dm_mat_type aijkokkos
[17]PETSC ERROR: -dm_plex_box_faces 4,4,4
[17]PETSC ERROR: -dm_plex_box_lower 0,0,0
[17]PETSC ERROR: -dm_plex_box_upper 2,2,2
[17]PETSC ERROR: -dm_plex_dim 3
[17]PETSC ERROR: -dm_plex_simplex 0
[17]PETSC ERROR: -dm_refine 4
[17]PETSC ERROR: -dm_vec_type kokkos
[17]PETSC ERROR: -dm_view
[17]PETSC ERROR: -ksp_converged_reason
[17]PETSC ERROR: -ksp_max_it 200
[17]PETSC ERROR: -ksp_norm_type unpreconditioned
[17]PETSC ERROR: -ksp_rtol 1.e-12
[17]PETSC ERROR: -ksp_type cg
[17]PETSC ERROR: -log_view
[17]PETSC ERROR: -mat_type aijkokkos
[17]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[17]PETSC ERROR: -mg_levels_ksp_type chebyshev
[17]PETSC ERROR: -mg_levels_pc_type jacobi
[17]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[17]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[28]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[28]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[28]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[28]PETSC ERROR: #14 main() at ex13.c:178
[28]PETSC ERROR: PETSc Option Table entries:
[28]PETSC ERROR: -benchmark_it 10
[28]PETSC ERROR: -dm_distribute
[28]PETSC ERROR: -dm_mat_type aijkokkos
[28]PETSC ERROR: -dm_plex_box_faces 4,4,4
[28]PETSC ERROR: -dm_plex_box_lower 0,0,0
[28]PETSC ERROR: -dm_plex_box_upper 2,2,2
[28]PETSC ERROR: -dm_plex_dim 3
[28]PETSC ERROR: -dm_plex_simplex 0
[28]PETSC ERROR: -dm_refine 4
[28]PETSC ERROR: -dm_vec_type kokkos
[28]PETSC ERROR: -dm_view
[28]PETSC ERROR: -ksp_converged_reason
[28]PETSC ERROR: -ksp_max_it 200
[28]PETSC ERROR: -ksp_norm_type unpreconditioned
[28]PETSC ERROR: -ksp_rtol 1.e-12
[28]PETSC ERROR: -ksp_type cg
[28]PETSC ERROR: -log_view
[45]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[45]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[45]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[28]PETSC ERROR: -mat_type aijkokkos
[28]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[28]PETSC ERROR: -mg_levels_ksp_type chebyshev
[28]PETSC ERROR: -mg_levels_pc_type jacobi
[28]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[28]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[28]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[28]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[45]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[45]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[45]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[45]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[45]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[45]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[45]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[45]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[45]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[30]PETSC ERROR: -dm_distribute
[30]PETSC ERROR: -dm_mat_type aijkokkos
[30]PETSC ERROR: -dm_plex_box_faces 4,4,4
[30]PETSC ERROR: -dm_plex_box_lower 0,0,0
[30]PETSC ERROR: -dm_plex_box_upper 2,2,2
[30]PETSC ERROR: -dm_plex_dim 3
[30]PETSC ERROR: -dm_plex_simplex 0
[30]PETSC ERROR: -dm_refine 4
[30]PETSC ERROR: -dm_vec_type kokkos
[30]PETSC ERROR: -dm_view
[30]PETSC ERROR: -ksp_converged_reason
[30]PETSC ERROR: -ksp_max_it 200
[30]PETSC ERROR: -ksp_norm_type unpreconditioned
[30]PETSC ERROR: -ksp_rtol 1.e-12
[30]PETSC ERROR: -ksp_type cg
[30]PETSC ERROR: -log_view
[30]PETSC ERROR: -mat_type aijkokkos
[30]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[30]PETSC ERROR: -mg_levels_ksp_type chebyshev
[30]PETSC ERROR: -mg_levels_pc_type jacobi
[30]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[30]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[30]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[30]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[30]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[45]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[45]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[45]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[45]PETSC ERROR: #14 main() at ex13.c:178
[45]PETSC ERROR: PETSc Option Table entries:
[45]PETSC ERROR: -benchmark_it 10
[45]PETSC ERROR: -dm_distribute
[45]PETSC ERROR: -dm_mat_type aijkokkos
[45]PETSC ERROR: -dm_plex_box_faces 4,4,4
[45]PETSC ERROR: -dm_plex_box_lower 0,0,0
[45]PETSC ERROR: -dm_plex_box_upper 2,2,2
[45]PETSC ERROR: -dm_plex_dim 3
[45]PETSC ERROR: -dm_plex_simplex 0
[45]PETSC ERROR: -dm_refine 4
[45]PETSC ERROR: -dm_vec_type kokkos
[45]PETSC ERROR: -dm_view
[45]PETSC ERROR: -ksp_converged_reason
[45]PETSC ERROR: -ksp_max_it 200
[45]PETSC ERROR: -ksp_norm_type unpreconditioned
[45]PETSC ERROR: -ksp_rtol 1.e-12
[45]PETSC ERROR: -ksp_type cg
[45]PETSC ERROR: -log_view
[30]PETSC ERROR: -pc_gamg_process_eq_limit 400
[30]PETSC ERROR: -pc_gamg_repartition false
[30]PETSC ERROR: -pc_gamg_reuse_interpolation true
[30]PETSC ERROR: -pc_gamg_threshold 0.01
[30]PETSC ERROR: -pc_type gamg
[30]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[30]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[30]PETSC ERROR: -petscpartitioner_type simple
[30]PETSC ERROR: -potential_petscspace_degree 2
[30]PETSC ERROR: -snes_lag_jacobian -2
[30]PETSC ERROR: -snes_max_it 1
[30]PETSC ERROR: -snes_rtol 1.e-8
[30]PETSC ERROR: -snes_type ksponly
[30]PETSC ERROR: -use_gpu_aware_mpi 0
[30]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 30] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[45]PETSC ERROR: -mat_type aijkokkos
[45]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[45]PETSC ERROR: -mg_levels_ksp_type chebyshev
[45]PETSC ERROR: -mg_levels_pc_type jacobi
[45]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[45]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[45]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[45]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[31]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[31]PETSC ERROR: No support for this operation for this object type
[31]PETSC ERROR: No method productsymbolic for Mat of type (null)
[31]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[31]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[31]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[31]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[31]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[31]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[31]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[46]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[46]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[46]PETSC ERROR: #14 main() at ex13.c:178
[46]PETSC ERROR: PETSc Option Table entries:
[46]PETSC ERROR: -benchmark_it 10
[46]PETSC ERROR: -dm_distribute
[46]PETSC ERROR: -dm_mat_type aijkokkos
[46]PETSC ERROR: -dm_plex_box_faces 4,4,4
[46]PETSC ERROR: -dm_plex_box_lower 0,0,0
[46]PETSC ERROR: -dm_plex_box_upper 2,2,2
[46]PETSC ERROR: -dm_plex_dim 3
[46]PETSC ERROR: -dm_plex_simplex 0
[46]PETSC ERROR: -dm_refine 4
[46]PETSC ERROR: -dm_vec_type kokkos
[46]PETSC ERROR: -dm_view
[46]PETSC ERROR: -ksp_converged_reason
[46]PETSC ERROR: -ksp_max_it 200
[46]PETSC ERROR: -ksp_norm_type unpreconditioned
[46]PETSC ERROR: -ksp_rtol 1.e-12
[46]PETSC ERROR: -ksp_type cg
[46]PETSC ERROR: -log_view
[46]PETSC ERROR: -mat_type aijkokkos
[46]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[17]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[17]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[17]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[17]PETSC ERROR: -pc_gamg_process_eq_limit 400
[17]PETSC ERROR: -pc_gamg_repartition false
[17]PETSC ERROR: -pc_gamg_reuse_interpolation true
[17]PETSC ERROR: -pc_gamg_threshold 0.01
[17]PETSC ERROR: -pc_type gamg
[17]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[17]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[17]PETSC ERROR: -petscpartitioner_type simple
[17]PETSC ERROR: -potential_petscspace_degree 2
[17]PETSC ERROR: -snes_lag_jacobian -2
[17]PETSC ERROR: -snes_max_it 1
[17]PETSC ERROR: -snes_rtol 1.e-8
[17]PETSC ERROR: -snes_type ksponly
[17]PETSC ERROR: -use_gpu_aware_mpi 0
[17]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
[31]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[31]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[31]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[46]PETSC ERROR: -mg_levels_ksp_type chebyshev
[46]PETSC ERROR: -mg_levels_pc_type jacobi
[46]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[46]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[46]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[46]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[46]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[46]PETSC ERROR: -pc_gamg_process_eq_limit 400
[46]PETSC ERROR: -pc_gamg_repartition false
[46]PETSC ERROR: -pc_gamg_reuse_interpolation true
[46]PETSC ERROR: -pc_gamg_threshold 0.01
[46]PETSC ERROR: -pc_type gamg
[46]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[46]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[46]PETSC ERROR: -petscpartitioner_type simple
[46]PETSC ERROR: -potential_petscspace_degree 2
[46]PETSC ERROR: -snes_lag_jacobian -2
[46]PETSC ERROR: -snes_max_it 1
[46]PETSC ERROR: -snes_rtol 1.e-8
[46]PETSC ERROR: -snes_type ksponly
[46]PETSC ERROR: -use_gpu_aware_mpi 0
MPICH ERROR [Rank 17] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[46]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 46] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[18]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: No support for this operation for this object type
[18]PETSC ERROR: No method productsymbolic for Mat of type (null)
[18]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[18]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[18]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[18]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[18]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[18]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[18]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[47]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[47]PETSC ERROR: No support for this operation for this object type
[47]PETSC ERROR: No method productsymbolic for Mat of type (null)
[47]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[47]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[47]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[47]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[47]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[47]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[47]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[18]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[18]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[18]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[47]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001125 by madams Sun Oct 30 05:30:05 2022
[47]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[47]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[18]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[18]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[18]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[18]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[18]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[18]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[18]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[18]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[18]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[18]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[18]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[18]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[18]PETSC ERROR: #14 main() at ex13.c:178
[18]PETSC ERROR: PETSc Option Table entries:
[18]PETSC ERROR: -benchmark_it 10
[18]PETSC ERROR: -dm_distribute
[18]PETSC ERROR: -dm_mat_type aijkokkos
[18]PETSC ERROR: -dm_plex_box_faces 4,4,4
[18]PETSC ERROR: -dm_plex_box_lower 0,0,0
[18]PETSC ERROR: -dm_plex_box_upper 2,2,2
[18]PETSC ERROR: -dm_plex_dim 3
[18]PETSC ERROR: -dm_plex_simplex 0
[18]PETSC ERROR: -dm_refine 4
[18]PETSC ERROR: -dm_vec_type kokkos
[19]PETSC ERROR: PETSc Option Table entries:
[19]PETSC ERROR: -benchmark_it 10
[19]PETSC ERROR: -dm_distribute
[19]PETSC ERROR: -dm_mat_type aijkokkos
[19]PETSC ERROR: -dm_plex_box_faces 4,4,4
[19]PETSC ERROR: -dm_plex_box_lower 0,0,0
[19]PETSC ERROR: -dm_plex_box_upper 2,2,2
[19]PETSC ERROR: -dm_plex_dim 3
[19]PETSC ERROR: -dm_plex_simplex 0
[19]PETSC ERROR: -dm_refine 4
[19]PETSC ERROR: -dm_vec_type kokkos
[19]PETSC ERROR: -dm_view
[19]PETSC ERROR: -ksp_converged_reason
[19]PETSC ERROR: -ksp_max_it 200
[19]PETSC ERROR: -ksp_norm_type unpreconditioned
[19]PETSC ERROR: -ksp_rtol 1.e-12
[19]PETSC ERROR: -ksp_type cg
[19]PETSC ERROR: -log_view
[19]PETSC ERROR: -mat_type aijkokkos
[19]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[19]PETSC ERROR: -mg_levels_ksp_type chebyshev
[19]PETSC ERROR: -mg_levels_pc_type jacobi
[19]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[19]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[19]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[19]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[19]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[19]PETSC ERROR: -pc_gamg_process_eq_limit 400
[19]PETSC ERROR: -pc_gamg_repartition false
[19]PETSC ERROR: -pc_gamg_reuse_interpolation true
[19]PETSC ERROR: -pc_gamg_threshold 0.01
[19]PETSC ERROR: -pc_type gamg
[19]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[19]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[19]PETSC ERROR: -petscpartitioner_type simple
[19]PETSC ERROR: -potential_petscspace_degree 2
[19]PETSC ERROR: -snes_lag_jacobian -2
[19]PETSC ERROR: -snes_max_it 1
[19]PETSC ERROR: -snes_rtol 1.e-8
[19]PETSC ERROR: -snes_type ksponly
[19]PETSC ERROR: -use_gpu_aware_mpi 0
[19]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 19] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[31]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[31]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[31]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[31]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[31]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[31]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[31]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[31]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[31]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[47]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[47]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[47]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[47]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[47]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[47]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[47]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[47]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[47]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[31]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[31]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[31]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[31]PETSC ERROR: #14 main() at ex13.c:178
[31]PETSC ERROR: PETSc Option Table entries:
[31]PETSC ERROR: -benchmark_it 10
[31]PETSC ERROR: -dm_distribute
[31]PETSC ERROR: -dm_mat_type aijkokkos
[31]PETSC ERROR: -dm_plex_box_faces 4,4,4
[31]PETSC ERROR: -dm_plex_box_lower 0,0,0
[31]PETSC ERROR: -dm_plex_box_upper 2,2,2
[31]PETSC ERROR: -dm_plex_dim 3
[31]PETSC ERROR: -dm_plex_simplex 0
[31]PETSC ERROR: -dm_refine 4
[31]PETSC ERROR: -dm_vec_type kokkos
[31]PETSC ERROR: -dm_view
[31]PETSC ERROR: -ksp_converged_reason
[31]PETSC ERROR: -ksp_max_it 200
[31]PETSC ERROR: -ksp_norm_type unpreconditioned
[31]PETSC ERROR: -ksp_rtol 1.e-12
[31]PETSC ERROR: -ksp_type cg
[31]PETSC ERROR: -log_view
[47]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[47]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[47]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[47]PETSC ERROR: #14 main() at ex13.c:178
[47]PETSC ERROR: PETSc Option Table entries:
[47]PETSC ERROR: -benchmark_it 10
[47]PETSC ERROR: -dm_distribute
[47]PETSC ERROR: -dm_mat_type aijkokkos
[47]PETSC ERROR: -dm_plex_box_faces 4,4,4
[47]PETSC ERROR: -dm_plex_box_lower 0,0,0
[47]PETSC ERROR: -dm_plex_box_upper 2,2,2
[47]PETSC ERROR: -dm_plex_dim 3
[47]PETSC ERROR: -dm_plex_simplex 0
[47]PETSC ERROR: -dm_refine 4
[47]PETSC ERROR: -dm_vec_type kokkos
[47]PETSC ERROR: -dm_view
[47]PETSC ERROR: -ksp_converged_reason
[47]PETSC ERROR: -ksp_max_it 200
[47]PETSC ERROR: -ksp_norm_type unpreconditioned
[47]PETSC ERROR: -ksp_rtol 1.e-12
[47]PETSC ERROR: -ksp_type cg
[47]PETSC ERROR: -log_view
[47]PETSC ERROR: -mat_type aijkokkos
[47]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[47]PETSC ERROR: -mg_levels_ksp_type chebyshev
[47]PETSC ERROR: -mg_levels_pc_type jacobi
[47]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[47]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[47]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[47]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[40]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[40]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[40]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[40]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[40]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[40]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[40]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[40]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[40]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[40]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[40]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[40]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[40]PETSC ERROR: #14 main() at ex13.c:178
[40]PETSC ERROR: PETSc Option Table entries:
[40]PETSC ERROR: -benchmark_it 10
[40]PETSC ERROR: -dm_distribute
[40]PETSC ERROR: -dm_mat_type aijkokkos
[40]PETSC ERROR: -dm_plex_box_faces 4,4,4
[40]PETSC ERROR: -dm_plex_box_lower 0,0,0
[40]PETSC ERROR: -dm_plex_box_upper 2,2,2
[40]PETSC ERROR: -dm_plex_dim 3
[40]PETSC ERROR: -dm_plex_simplex 0
[40]PETSC ERROR: -dm_refine 4
[40]PETSC ERROR: -dm_vec_type kokkos
[40]PETSC ERROR: -dm_view
[40]PETSC ERROR: -ksp_converged_reason
[40]PETSC ERROR: -ksp_max_it 200
[40]PETSC ERROR: -ksp_norm_type unpreconditioned
[40]PETSC ERROR: -ksp_rtol 1.e-12
[40]PETSC ERROR: -ksp_type cg
[40]PETSC ERROR: -log_view
[40]PETSC ERROR: -mat_type aijkokkos
[40]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[40]PETSC ERROR: -mg_levels_ksp_type chebyshev
[40]PETSC ERROR: -mg_levels_pc_type jacobi
[40]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[40]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[40]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[40]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[40]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[40]PETSC ERROR: -pc_gamg_process_eq_limit 400
[40]PETSC ERROR: -pc_gamg_repartition false
[40]PETSC ERROR: -pc_gamg_reuse_interpolation true
[40]PETSC ERROR: -pc_gamg_threshold 0.01
[40]PETSC ERROR: -pc_type gamg
[40]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[40]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[40]PETSC ERROR: -petscpartitioner_type simple
[40]PETSC ERROR: -potential_petscspace_degree 2
[40]PETSC ERROR: -snes_lag_jacobian -2
[40]PETSC ERROR: -snes_max_it 1
[40]PETSC ERROR: -snes_rtol 1.e-8
[40]PETSC ERROR: -snes_type ksponly
[40]PETSC ERROR: -use_gpu_aware_mpi 0
[40]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 40] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[58]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[58]PETSC ERROR: -pc_gamg_process_eq_limit 400
[58]PETSC ERROR: -pc_gamg_repartition false
[58]PETSC ERROR: -pc_gamg_reuse_interpolation true
[58]PETSC ERROR: -pc_gamg_threshold 0.01
[58]PETSC ERROR: -pc_type gamg
[58]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[58]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[58]PETSC ERROR: -petscpartitioner_type simple
[58]PETSC ERROR: -potential_petscspace_degree 2
[58]PETSC ERROR: -snes_lag_jacobian -2
[58]PETSC ERROR: -snes_max_it 1
[58]PETSC ERROR: -snes_rtol 1.e-8
[58]PETSC ERROR: -snes_type ksponly
[58]PETSC ERROR: -use_gpu_aware_mpi 0
[58]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 58] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[60]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[60]PETSC ERROR: -pc_gamg_process_eq_limit 400
[60]PETSC ERROR: -pc_gamg_repartition false
[60]PETSC ERROR: -pc_gamg_reuse_interpolation true
[60]PETSC ERROR: -pc_gamg_threshold 0.01
[60]PETSC ERROR: -pc_type gamg
[60]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[60]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[60]PETSC ERROR: -petscpartitioner_type simple
[60]PETSC ERROR: -potential_petscspace_degree 2
[60]PETSC ERROR: -snes_lag_jacobian -2
[60]PETSC ERROR: -snes_max_it 1
[60]PETSC ERROR: -snes_rtol 1.e-8
[60]PETSC ERROR: -snes_type ksponly
[60]PETSC ERROR: -use_gpu_aware_mpi 0
[60]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 60] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001129] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[20]PETSC ERROR: -ksp_max_it 200
[20]PETSC ERROR: -ksp_norm_type unpreconditioned
[20]PETSC ERROR: -ksp_rtol 1.e-12
[20]PETSC ERROR: -ksp_type cg
[20]PETSC ERROR: -log_view
[20]PETSC ERROR: -mat_type aijkokkos
[20]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[20]PETSC ERROR: -mg_levels_ksp_type chebyshev
[20]PETSC ERROR: -mg_levels_pc_type jacobi
[20]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[20]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[20]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[20]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[20]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[20]PETSC ERROR: -pc_gamg_process_eq_limit 400
[20]PETSC ERROR: -pc_gamg_repartition false
[20]PETSC ERROR: -pc_gamg_reuse_interpolation true
[20]PETSC ERROR: -pc_gamg_threshold 0.01
[20]PETSC ERROR: -pc_type gamg
[20]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[20]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[20]PETSC ERROR: -petscpartitioner_type simple
[41]PETSC ERROR: -petscpartitioner_type simple
[41]PETSC ERROR: -potential_petscspace_degree 2
[41]PETSC ERROR: -snes_lag_jacobian -2
[41]PETSC ERROR: -snes_max_it 1
[41]PETSC ERROR: -snes_rtol 1.e-8
[41]PETSC ERROR: -snes_type ksponly
[41]PETSC ERROR: -use_gpu_aware_mpi 0
[41]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 41] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[20]PETSC ERROR: -potential_petscspace_degree 2
[20]PETSC ERROR: -snes_lag_jacobian -2
[20]PETSC ERROR: -snes_max_it 1
[20]PETSC ERROR: -snes_rtol 1.e-8
[20]PETSC ERROR: -snes_type ksponly
[20]PETSC ERROR: -use_gpu_aware_mpi 0
[20]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 20] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[42]PETSC ERROR: -dm_view
[42]PETSC ERROR: -ksp_converged_reason
[42]PETSC ERROR: -ksp_max_it 200
[42]PETSC ERROR: -ksp_norm_type unpreconditioned
[42]PETSC ERROR: -ksp_rtol 1.e-12
[42]PETSC ERROR: -ksp_type cg
[42]PETSC ERROR: -log_view
[42]PETSC ERROR: -mat_type aijkokkos
[42]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[42]PETSC ERROR: -mg_levels_ksp_type chebyshev
[42]PETSC ERROR: -mg_levels_pc_type jacobi
[42]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[42]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[42]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[42]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[42]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[42]PETSC ERROR: -pc_gamg_process_eq_limit 400
[42]PETSC ERROR: -pc_gamg_repartition false
[42]PETSC ERROR: -pc_gamg_reuse_interpolation true
[42]PETSC ERROR: -pc_gamg_threshold 0.01
[42]PETSC ERROR: -pc_type gamg
[42]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[42]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[21]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: No support for this operation for this object type
[21]PETSC ERROR: No method productsymbolic for Mat of type (null)
[21]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[21]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[21]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[21]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[21]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[21]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[21]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[35]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[35]PETSC ERROR: -pc_gamg_process_eq_limit 400
[35]PETSC ERROR: -pc_gamg_repartition false
[35]PETSC ERROR: -pc_gamg_reuse_interpolation true
[35]PETSC ERROR: -pc_gamg_threshold 0.01
[35]PETSC ERROR: -pc_type gamg
[35]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[35]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[35]PETSC ERROR: -petscpartitioner_type simple
[35]PETSC ERROR: -potential_petscspace_degree 2
[35]PETSC ERROR: -snes_lag_jacobian -2
[35]PETSC ERROR: -snes_max_it 1
[35]PETSC ERROR: -snes_rtol 1.e-8
[35]PETSC ERROR: -snes_type ksponly
[35]PETSC ERROR: -use_gpu_aware_mpi 0
[35]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 35] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[21]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001120 by madams Sun Oct 30 05:30:05 2022
[21]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[21]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[36]PETSC ERROR: -snes_type ksponly
[36]PETSC ERROR: -use_gpu_aware_mpi 0
[36]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 36] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[21]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[21]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[21]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[21]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[21]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[21]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[21]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[21]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[21]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[37]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[37]PETSC ERROR: No support for this operation for this object type
[37]PETSC ERROR: No method productsymbolic for Mat of type (null)
[37]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[37]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[37]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[37]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[37]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[37]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[37]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
[21]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[21]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[21]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[21]PETSC ERROR: #14 main() at ex13.c:178
[21]PETSC ERROR: PETSc Option Table entries:
[21]PETSC ERROR: -benchmark_it 10
[21]PETSC ERROR: -dm_distribute
[21]PETSC ERROR: -dm_mat_type aijkokkos
[21]PETSC ERROR: -dm_plex_box_faces 4,4,4
[21]PETSC ERROR: -dm_plex_box_lower 0,0,0
[21]PETSC ERROR: -dm_plex_box_upper 2,2,2
[21]PETSC ERROR: -dm_plex_dim 3
[21]PETSC ERROR: -dm_plex_simplex 0
[21]PETSC ERROR: -dm_refine 4
[21]PETSC ERROR: -dm_vec_type kokkos
[21]PETSC ERROR: -dm_view
[21]PETSC ERROR: -ksp_converged_reason
[21]PETSC ERROR: -ksp_max_it 200
[21]PETSC ERROR: -ksp_norm_type unpreconditioned
[21]PETSC ERROR: -ksp_rtol 1.e-12
[21]PETSC ERROR: -ksp_type cg
[21]PETSC ERROR: -log_view
[37]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001124 by madams Sun Oct 30 05:30:05 2022
[37]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[37]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
[21]PETSC ERROR: -mat_type aijkokkos
[21]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[21]PETSC ERROR: -mg_levels_ksp_type chebyshev
[21]PETSC ERROR: -mg_levels_pc_type jacobi
[21]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[21]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[21]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[21]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[37]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[37]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[37]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[37]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[37]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[37]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[37]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[37]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[37]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[37]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[37]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[37]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[37]PETSC ERROR: #14 main() at ex13.c:178
[37]PETSC ERROR: PETSc Option Table entries:
[37]PETSC ERROR: -benchmark_it 10
[37]PETSC ERROR: -dm_distribute
[37]PETSC ERROR: -dm_mat_type aijkokkos
[37]PETSC ERROR: -dm_plex_box_faces 4,4,4
[37]PETSC ERROR: -dm_plex_box_lower 0,0,0
[37]PETSC ERROR: -dm_plex_box_upper 2,2,2
[37]PETSC ERROR: -dm_plex_dim 3
[37]PETSC ERROR: -dm_plex_simplex 0
[37]PETSC ERROR: -dm_refine 4
[37]PETSC ERROR: -dm_vec_type kokkos
[37]PETSC ERROR: -dm_view
[37]PETSC ERROR: -ksp_converged_reason
[37]PETSC ERROR: -ksp_max_it 200
[37]PETSC ERROR: -ksp_norm_type unpreconditioned
[37]PETSC ERROR: -ksp_rtol 1.e-12
[37]PETSC ERROR: -ksp_type cg
[37]PETSC ERROR: -log_view
[37]PETSC ERROR: -mat_type aijkokkos
[37]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[37]PETSC ERROR: -mg_levels_ksp_type chebyshev
[37]PETSC ERROR: -mg_levels_pc_type jacobi
[37]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[37]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[37]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[37]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[32]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[32]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[32]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[32]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[32]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[32]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[32]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[32]PETSC ERROR: #14 main() at ex13.c:178
[32]PETSC ERROR: PETSc Option Table entries:
[32]PETSC ERROR: -benchmark_it 10
[32]PETSC ERROR: -dm_distribute
[32]PETSC ERROR: -dm_mat_type aijkokkos
[32]PETSC ERROR: -dm_plex_box_faces 4,4,4
[32]PETSC ERROR: -dm_plex_box_lower 0,0,0
[32]PETSC ERROR: -dm_plex_box_upper 2,2,2
[32]PETSC ERROR: -dm_plex_dim 3
[32]PETSC ERROR: -dm_plex_simplex 0
[32]PETSC ERROR: -dm_refine 4
[32]PETSC ERROR: -dm_vec_type kokkos
[32]PETSC ERROR: -dm_view
[32]PETSC ERROR: -ksp_converged_reason
[32]PETSC ERROR: -ksp_max_it 200
[32]PETSC ERROR: -ksp_norm_type unpreconditioned
[32]PETSC ERROR: -ksp_rtol 1.e-12
[32]PETSC ERROR: -ksp_type cg
[32]PETSC ERROR: -log_view
[32]PETSC ERROR: -mat_type aijkokkos
[32]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[32]PETSC ERROR: -mg_levels_ksp_type chebyshev
[32]PETSC ERROR: -mg_levels_pc_type jacobi
[32]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[32]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[32]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[32]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[32]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[32]PETSC ERROR: -pc_gamg_process_eq_limit 400
[32]PETSC ERROR: -pc_gamg_repartition false
[32]PETSC ERROR: -pc_gamg_reuse_interpolation true
[32]PETSC ERROR: -pc_gamg_threshold 0.01
[32]PETSC ERROR: -pc_type gamg
[32]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[32]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[32]PETSC ERROR: -petscpartitioner_type simple
[32]PETSC ERROR: -potential_petscspace_degree 2
[32]PETSC ERROR: -snes_lag_jacobian -2
[32]PETSC ERROR: -snes_max_it 1
[32]PETSC ERROR: -snes_rtol 1.e-8
[32]PETSC ERROR: -snes_type ksponly
[32]PETSC ERROR: -use_gpu_aware_mpi 0
[32]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 32] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[34]PETSC ERROR: -ksp_norm_type unpreconditioned
[34]PETSC ERROR: -ksp_rtol 1.e-12
[34]PETSC ERROR: -ksp_type cg
[34]PETSC ERROR: -log_view
[34]PETSC ERROR: -mat_type aijkokkos
[34]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[34]PETSC ERROR: -mg_levels_ksp_type chebyshev
[34]PETSC ERROR: -mg_levels_pc_type jacobi
[34]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[34]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[34]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[34]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[34]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[34]PETSC ERROR: -pc_gamg_process_eq_limit 400
[34]PETSC ERROR: -pc_gamg_repartition false
[34]PETSC ERROR: -pc_gamg_reuse_interpolation true
[34]PETSC ERROR: -pc_gamg_threshold 0.01
[34]PETSC ERROR: -pc_type gamg
[34]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[34]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[34]PETSC ERROR: -petscpartitioner_type simple
[34]PETSC ERROR: -potential_petscspace_degree 2
[34]PETSC ERROR: -snes_lag_jacobian -2
[34]PETSC ERROR: -snes_max_it 1
[34]PETSC ERROR: -snes_rtol 1.e-8
[34]PETSC ERROR: -snes_type ksponly
[34]PETSC ERROR: -use_gpu_aware_mpi 0
[34]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 34] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
MPICH ERROR [Rank 23] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[16]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[16]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[16]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[16]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[16]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[16]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[16]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[16]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[16]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[16]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[16]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[16]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[16]PETSC ERROR: #14 main() at ex13.c:178
[16]PETSC ERROR: PETSc Option Table entries:
[16]PETSC ERROR: -benchmark_it 10
[16]PETSC ERROR: -dm_distribute
[16]PETSC ERROR: -dm_mat_type aijkokkos
[16]PETSC ERROR: -dm_plex_box_faces 4,4,4
[16]PETSC ERROR: -dm_plex_box_lower 0,0,0
[16]PETSC ERROR: -dm_plex_box_upper 2,2,2
[16]PETSC ERROR: -dm_plex_dim 3
[16]PETSC ERROR: -dm_plex_simplex 0
[16]PETSC ERROR: -dm_refine 4
[16]PETSC ERROR: -dm_vec_type kokkos
[16]PETSC ERROR: -dm_view
[16]PETSC ERROR: -ksp_converged_reason
[16]PETSC ERROR: -ksp_max_it 200
[16]PETSC ERROR: -ksp_norm_type unpreconditioned
[16]PETSC ERROR: -ksp_rtol 1.e-12
[16]PETSC ERROR: -ksp_type cg
[16]PETSC ERROR: -log_view
[16]PETSC ERROR: -mat_type aijkokkos
[16]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[16]PETSC ERROR: -mg_levels_ksp_type chebyshev
[16]PETSC ERROR: -mg_levels_pc_type jacobi
[16]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[16]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[16]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[16]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[16]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[16]PETSC ERROR: -pc_gamg_process_eq_limit 400
[16]PETSC ERROR: -pc_gamg_repartition false
[16]PETSC ERROR: -pc_gamg_reuse_interpolation true
[16]PETSC ERROR: -pc_gamg_threshold 0.01
[16]PETSC ERROR: -pc_type gamg
[16]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[16]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[16]PETSC ERROR: -petscpartitioner_type simple
[16]PETSC ERROR: -potential_petscspace_degree 2
[16]PETSC ERROR: -snes_lag_jacobian -2
[16]PETSC ERROR: -snes_max_it 1
[16]PETSC ERROR: -snes_rtol 1.e-8
[16]PETSC ERROR: -snes_type ksponly
[31]PETSC ERROR: -mat_type aijkokkos
[31]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[31]PETSC ERROR: -mg_levels_ksp_type chebyshev
[31]PETSC ERROR: -mg_levels_pc_type jacobi
[31]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[31]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[31]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[31]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[42]PETSC ERROR: -petscpartitioner_type simple
[42]PETSC ERROR: -potential_petscspace_degree 2
[42]PETSC ERROR: -snes_lag_jacobian -2
[42]PETSC ERROR: -snes_max_it 1
[42]PETSC ERROR: -snes_rtol 1.e-8
[42]PETSC ERROR: -snes_type ksponly
[42]PETSC ERROR: -use_gpu_aware_mpi 0
[42]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 42] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[16]PETSC ERROR: -use_gpu_aware_mpi 0
[16]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 16] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[24]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[24]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[24]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[24]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[24]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
[24]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[24]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[24]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[24]PETSC ERROR: #14 main() at ex13.c:178
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -benchmark_it 10
[24]PETSC ERROR: -dm_distribute
[24]PETSC ERROR: -dm_mat_type aijkokkos
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[24]PETSC ERROR: -dm_plex_box_faces 4,4,4
[24]PETSC ERROR: -dm_plex_box_lower 0,0,0
[24]PETSC ERROR: -dm_plex_box_upper 2,2,2
[24]PETSC ERROR: -dm_plex_dim 3
[24]PETSC ERROR: -dm_plex_simplex 0
[24]PETSC ERROR: -dm_refine 4
[24]PETSC ERROR: -dm_vec_type kokkos
[24]PETSC ERROR: -dm_view
[24]PETSC ERROR: -ksp_converged_reason
[24]PETSC ERROR: -ksp_max_it 200
[24]PETSC ERROR: -ksp_norm_type unpreconditioned
[24]PETSC ERROR: -ksp_rtol 1.e-12
[24]PETSC ERROR: -ksp_type cg
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mat_type aijkokkos
[24]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[24]PETSC ERROR: -mg_levels_ksp_type chebyshev
[24]PETSC ERROR: -mg_levels_pc_type jacobi
[24]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[24]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[24]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[24]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[24]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[24]PETSC ERROR: -pc_gamg_process_eq_limit 400
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[18]PETSC ERROR: -dm_view
[18]PETSC ERROR: -ksp_converged_reason
[18]PETSC ERROR: -ksp_max_it 200
[18]PETSC ERROR: -ksp_norm_type unpreconditioned
[18]PETSC ERROR: -ksp_rtol 1.e-12
[18]PETSC ERROR: -ksp_type cg
[18]PETSC ERROR: -log_view
[18]PETSC ERROR: -mat_type aijkokkos
[18]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[18]PETSC ERROR: -mg_levels_ksp_type chebyshev
[18]PETSC ERROR: -mg_levels_pc_type jacobi
[18]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[18]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[18]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[18]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
[18]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[18]PETSC ERROR: -pc_gamg_process_eq_limit 400
[18]PETSC ERROR: -pc_gamg_repartition false
[18]PETSC ERROR: -pc_gamg_reuse_interpolation true
[18]PETSC ERROR: -pc_gamg_threshold 0.01
[18]PETSC ERROR: -pc_type gamg
[18]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[18]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[24]PETSC ERROR: -pc_gamg_repartition false
[24]PETSC ERROR: -pc_gamg_reuse_interpolation true
[24]PETSC ERROR: -pc_gamg_threshold 0.01
[24]PETSC ERROR: -pc_type gamg
[24]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[24]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[24]PETSC ERROR: -petscpartitioner_type simple
[24]PETSC ERROR: -potential_petscspace_degree 2
[24]PETSC ERROR: -snes_lag_jacobian -2
[24]PETSC ERROR: -snes_max_it 1
[24]PETSC ERROR: -snes_rtol 1.e-8
[24]PETSC ERROR: -snes_type ksponly
[24]PETSC ERROR: -use_gpu_aware_mpi 0
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 24] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[18]PETSC ERROR: -petscpartitioner_type simple
[18]PETSC ERROR: -potential_petscspace_degree 2
[18]PETSC ERROR: -snes_lag_jacobian -2
[18]PETSC ERROR: -snes_max_it 1
[18]PETSC ERROR: -snes_rtol 1.e-8
[18]PETSC ERROR: -snes_type ksponly
[18]PETSC ERROR: -use_gpu_aware_mpi 0
[18]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 18] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[25]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 25] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[27]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[27]PETSC ERROR: -pc_gamg_process_eq_limit 400
[27]PETSC ERROR: -pc_gamg_repartition false
[27]PETSC ERROR: -pc_gamg_reuse_interpolation true
[27]PETSC ERROR: -pc_gamg_threshold 0.01
[27]PETSC ERROR: -pc_type gamg
[27]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[27]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[27]PETSC ERROR: -petscpartitioner_type simple
[27]PETSC ERROR: -potential_petscspace_degree 2
[27]PETSC ERROR: -snes_lag_jacobian -2
[27]PETSC ERROR: -snes_max_it 1
[27]PETSC ERROR: -snes_rtol 1.e-8
[27]PETSC ERROR: -snes_type ksponly
[27]PETSC ERROR: -use_gpu_aware_mpi 0
[27]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 27] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[37]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[37]PETSC ERROR: -pc_gamg_process_eq_limit 400
[37]PETSC ERROR: -pc_gamg_repartition false
[37]PETSC ERROR: -pc_gamg_reuse_interpolation true
[37]PETSC ERROR: -pc_gamg_threshold 0.01
[37]PETSC ERROR: -pc_type gamg
[37]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[37]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[37]PETSC ERROR: -petscpartitioner_type simple
[37]PETSC ERROR: -potential_petscspace_degree 2
[37]PETSC ERROR: -snes_lag_jacobian -2
[37]PETSC ERROR: -snes_max_it 1
[37]PETSC ERROR: -snes_rtol 1.e-8
[37]PETSC ERROR: -snes_type ksponly
[37]PETSC ERROR: -use_gpu_aware_mpi 0
[37]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 37] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001124] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
[28]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[28]PETSC ERROR: -pc_gamg_process_eq_limit 400
[28]PETSC ERROR: -pc_gamg_repartition false
[28]PETSC ERROR: -pc_gamg_reuse_interpolation true
[28]PETSC ERROR: -pc_gamg_threshold 0.01
[28]PETSC ERROR: -pc_type gamg
[28]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[28]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[28]PETSC ERROR: -petscpartitioner_type simple
[28]PETSC ERROR: -potential_petscspace_degree 2
[28]PETSC ERROR: -snes_lag_jacobian -2
[28]PETSC ERROR: -snes_max_it 1
[28]PETSC ERROR: -snes_rtol 1.e-8
[28]PETSC ERROR: -snes_type ksponly
[28]PETSC ERROR: -use_gpu_aware_mpi 0
[28]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 28] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[29]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[29]PETSC ERROR: No support for this operation for this object type
[29]PETSC ERROR: No method productsymbolic for Mat of type (null)
[29]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[29]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[29]PETSC ERROR: Option left: name:-mg_levels_ksp_chebyshev_esteig value: 0,0.05,0,1.05
[29]PETSC ERROR: Option left: name:-mg_levels_ksp_type value: chebyshev
[29]PETSC ERROR: Option left: name:-mg_levels_pc_type value: jacobi
[29]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[29]PETSC ERROR: Petsc Development GIT revision: v3.18.1-122-g50ed725bd85 GIT Date: 2022-10-28 19:54:01 +0000
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[29]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001121 by madams Sun Oct 30 05:30:05 2022
[29]PETSC ERROR: Configure options --CFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CXXFLAGS=" -g -DPETSC_HAVE_KOKKOS_KERNELS_GMRES" --CUDAFLAGS="-g -Xcompiler -rdynamic" --with-cc=cc --with-cxx=CC --with-fc=ftn --download-hypre=1 --download-amgx --with-fortran-bindings=0 --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
[29]PETSC ERROR: #1 MatProductSymbolic_MPIAIJKokkos_AB() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[29]PETSC ERROR: #2 MatProductSymbolic_MPIAIJKokkos() at /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
[29]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:793
[29]PETSC ERROR: #4 MatProduct_Private() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9820
[29]PETSC ERROR: #5 MatMatMult() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9897
[29]PETSC ERROR: #6 PCGAMGOptProlongator_AGG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/agg.c:769
[29]PETSC ERROR: #7 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:639
[29]PETSC ERROR: #8 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:994
[29]PETSC ERROR: #9 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
[29]PETSC ERROR: #10 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:825
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[29]PETSC ERROR: #11 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1071
[29]PETSC ERROR: #12 SNESSolve_KSPONLY() at /global/u2/m/madams/petsc/src/snes/impls/ksponly/ksponly.c:48
[29]PETSC ERROR: #13 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4689
[29]PETSC ERROR: #14 main() at ex13.c:178
[29]PETSC ERROR: PETSc Option Table entries:
[29]PETSC ERROR: -benchmark_it 10
[29]PETSC ERROR: -dm_distribute
[29]PETSC ERROR: -dm_mat_type aijkokkos
[29]PETSC ERROR: -dm_plex_box_faces 4,4,4
[29]PETSC ERROR: -dm_plex_box_lower 0,0,0
[29]PETSC ERROR: -dm_plex_box_upper 2,2,2
[29]PETSC ERROR: -dm_plex_dim 3
[29]PETSC ERROR: -dm_plex_simplex 0
[29]PETSC ERROR: -dm_refine 4
[29]PETSC ERROR: -dm_vec_type kokkos
[29]PETSC ERROR: -dm_view
[29]PETSC ERROR: -ksp_converged_reason
[29]PETSC ERROR: -ksp_max_it 200
[29]PETSC ERROR: -ksp_norm_type unpreconditioned
[29]PETSC ERROR: -ksp_rtol 1.e-12
[29]PETSC ERROR: -ksp_type cg
[29]PETSC ERROR: -log_view
[29]PETSC ERROR: -mat_type aijkokkos
[29]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
[29]PETSC ERROR: -mg_levels_ksp_type chebyshev
[29]PETSC ERROR: -mg_levels_pc_type jacobi
[29]PETSC ERROR: -pc_gamg_aggressive_coarsening 1
[29]PETSC ERROR: -pc_gamg_coarse_eq_limit 100
[29]PETSC ERROR: -pc_gamg_coarse_grid_layout_type compact
[29]PETSC ERROR: -pc_gamg_esteig_ksp_max_it 10
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[31]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[31]PETSC ERROR: -pc_gamg_process_eq_limit 400
[31]PETSC ERROR: -pc_gamg_repartition false
[31]PETSC ERROR: -pc_gamg_reuse_interpolation true
[31]PETSC ERROR: -pc_gamg_threshold 0.01
[31]PETSC ERROR: -pc_type gamg
[31]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[31]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[31]PETSC ERROR: -petscpartitioner_type simple
[31]PETSC ERROR: -potential_petscspace_degree 2
[31]PETSC ERROR: -snes_lag_jacobian -2
[31]PETSC ERROR: -snes_max_it 1
[31]PETSC ERROR: -snes_rtol 1.e-8
[31]PETSC ERROR: -snes_type ksponly
[31]PETSC ERROR: -use_gpu_aware_mpi 0
[31]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 31] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[29]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[29]PETSC ERROR: -pc_gamg_process_eq_limit 400
[29]PETSC ERROR: -pc_gamg_repartition false
[29]PETSC ERROR: -pc_gamg_reuse_interpolation true
[29]PETSC ERROR: -pc_gamg_threshold 0.01
[29]PETSC ERROR: -pc_type gamg
[29]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[29]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[29]PETSC ERROR: -petscpartitioner_type simple
[29]PETSC ERROR: -potential_petscspace_degree 2
[29]PETSC ERROR: -snes_lag_jacobian -2
[29]PETSC ERROR: -snes_max_it 1
[29]PETSC ERROR: -snes_rtol 1.e-8
[29]PETSC ERROR: -snes_type ksponly
[29]PETSC ERROR: -use_gpu_aware_mpi 0
[29]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 29] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001121] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[21]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[21]PETSC ERROR: -pc_gamg_process_eq_limit 400
[21]PETSC ERROR: -pc_gamg_repartition false
[21]PETSC ERROR: -pc_gamg_reuse_interpolation true
[21]PETSC ERROR: -pc_gamg_threshold 0.01
[21]PETSC ERROR: -pc_type gamg
[21]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[21]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[21]PETSC ERROR: -petscpartitioner_type simple
[21]PETSC ERROR: -potential_petscspace_degree 2
[21]PETSC ERROR: -snes_lag_jacobian -2
[21]PETSC ERROR: -snes_max_it 1
[21]PETSC ERROR: -snes_rtol 1.e-8
[21]PETSC ERROR: -snes_type ksponly
[21]PETSC ERROR: -use_gpu_aware_mpi 0
[21]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 21] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001120] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[45]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[45]PETSC ERROR: -pc_gamg_process_eq_limit 400
[45]PETSC ERROR: -pc_gamg_repartition false
[45]PETSC ERROR: -pc_gamg_reuse_interpolation true
[45]PETSC ERROR: -pc_gamg_threshold 0.01
[45]PETSC ERROR: -pc_type gamg
[45]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[45]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[45]PETSC ERROR: -petscpartitioner_type simple
[45]PETSC ERROR: -potential_petscspace_degree 2
[45]PETSC ERROR: -snes_lag_jacobian -2
[45]PETSC ERROR: -snes_max_it 1
[45]PETSC ERROR: -snes_rtol 1.e-8
[45]PETSC ERROR: -snes_type ksponly
[45]PETSC ERROR: -use_gpu_aware_mpi 0
[45]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 45] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[47]PETSC ERROR: -pc_gamg_esteig_ksp_type cg
[47]PETSC ERROR: -pc_gamg_process_eq_limit 400
[47]PETSC ERROR: -pc_gamg_repartition false
[47]PETSC ERROR: -pc_gamg_reuse_interpolation true
[47]PETSC ERROR: -pc_gamg_threshold 0.01
[47]PETSC ERROR: -pc_type gamg
[47]PETSC ERROR: -petscpartitioner_simple_node_grid 2,2,2
[47]PETSC ERROR: -petscpartitioner_simple_process_grid 2,2,2
[47]PETSC ERROR: -petscpartitioner_type simple
[47]PETSC ERROR: -potential_petscspace_degree 2
[47]PETSC ERROR: -snes_lag_jacobian -2
[47]PETSC ERROR: -snes_max_it 1
[47]PETSC ERROR: -snes_rtol 1.e-8
[47]PETSC ERROR: -snes_type ksponly
[47]PETSC ERROR: -use_gpu_aware_mpi 0
[47]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
MPICH ERROR [Rank 47] [job id 3522949.0] [Sun Oct 30 05:30:12 2022] [nid001125] - Abort(56) (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 56) - process 0
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
srun: error: nid001124: task 36: Exited with exit code 56
srun: launch/slurm: _step_signal: Terminating StepId=3522949.0
slurmstepd: error: *** STEP 3522949.0 ON nid001116 CANCELLED AT 2022-10-30T12:30:12 ***
srun: error: nid001125: tasks 45-46: Exited with exit code 56
srun: error: nid001128: task 54: Exited with exit code 56
srun: error: nid001121: tasks 25,27-28: Exited with exit code 56
srun: error: nid001120: task 18: Exited with exit code 56
srun: error: nid001116: tasks 2,4: Exited with exit code 56
srun: error: nid001129: task 63: Exited with exit code 56
srun: error: nid001117: tasks 9,11,13: Exited with exit code 56
srun: error: nid001124: tasks 35,37-38: Exited with exit code 56
srun: error: nid001128: tasks 50,55: Exited with exit code 56
srun: error: nid001120: tasks 16,19,22: Exited with exit code 56
srun: error: nid001129: tasks 61-62: Exited with exit code 56
srun: error: nid001125: tasks 41,44: Exited with exit code 56
srun: error: nid001116: tasks 1,5-6: Exited with exit code 56
srun: error: nid001128: task 52: Exited with exit code 56
srun: error: nid001121: task 29: Exited with exit code 56
srun: error: nid001129: tasks 57,59: Exited with exit code 56
srun: error: nid001124: task 33: Exited with exit code 56
srun: error: nid001117: tasks 12,15: Exited with exit code 56
srun: error: nid001125: tasks 43,47: Exited with exit code 56
srun: error: nid001128: tasks 51,53: Exited with exit code 56
srun: error: nid001121: tasks 26,31: Exited with exit code 56
srun: error: nid001120: tasks 17,20: Exited with exit code 56
srun: error: nid001124: tasks 32,39: Exited with exit code 56
srun: error: nid001116: tasks 0,3: Exited with exit code 56
srun: error: nid001129: tasks 56,60: Exited with exit code 56
srun: error: nid001120: task 23: Exited with exit code 56
srun: error: nid001128: task 48: Exited with exit code 56
srun: error: nid001121: task 30: Exited with exit code 56
srun: error: nid001125: task 42: Exited with exit code 56
srun: error: nid001117: tasks 8,14: Exited with exit code 56
srun: error: nid001125: task 40: Exited with exit code 56
srun: error: nid001120: task 21: Exited with exit code 56
srun: error: nid001128: task 49: Exited with exit code 56
srun: error: nid001124: task 34: Exited with exit code 56
srun: error: nid001129: task 58: Exited with exit code 56
srun: error: nid001117: task 10: Exited with exit code 56
srun: error: nid001121: task 24: Exited with exit code 56
srun: error: nid001116: task 7: Exited with exit code 56
+ date
Sun 30 Oct 2022 05:30:13 AM PDT
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 905439 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221116/72ffed1f/attachment-0001.obj>
More information about the petsc-users
mailing list