[petsc-users] [EXTERNAL] Re: Using multiple MPI ranks with COO interface crashes in some cases

Wed Nov 16 17:32:09 CST 2022

Junchao, I tried with the Kokkos dev branch and get this with 8 processes
(and the .petscrc file that I sent/appended):

 srun -n8 -N1 --gpus-per-task=1 --gpu-bind=closest ../ex13
-dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 2,2,2
-dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1
-dm_refine 4 -dm_view -log_tracexxx -log_view -dm_mat_type aijkokkos
-dm_vec_type kokkos

DM Object: box 8 MPI processes
  type: plex
box in 3 dimensions:
  Number of 0-cells per rank: 4913 4913 4913 4913 4913 4913 4913 4913
  Number of 1-cells per rank: 13872 13872 13872 13872 13872 13872 13872
13872
  Number of 2-cells per rank: 13056 13056 13056 13056 13056 13056 13056
13056
  Number of 3-cells per rank: 4096 4096 4096 4096 4096 4096 4096 4096
Labels:
  celltype: 4 strata with value/size (0 (4913), 1 (13872), 4 (13056), 7
(4096))
  depth: 4 strata with value/size (0 (4913), 1 (13872), 2 (13056), 3 (4096))
  marker: 1 strata with value/size (1 (3169))
  Face Sets: 3 strata with value/size (1 (961), 3 (961), 6 (961))
Number equations N = 250047
[5]PETSC ERROR:
------------------------------------------------------------------------
[5]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal
memory access
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[5]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
https://petsc.org/release/faq/
[5]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[5]PETSC ERROR: The line numbers in the error traceback are not always
exact.
[5]PETSC ERROR: #1 MPI function
[5]PETSC ERROR: #2 PetscSFLinkWaitRequests_MPI() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:53
[5]PETSC ERROR: #3 PetscSFLinkFinishCommunication() at
/gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:277
[5]PETSC ERROR: #4 PetscSFBcastEnd_Basic() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:205
[5]PETSC ERROR: #5 PetscSFBcastEnd() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/interface/sf.c:1477
[5]PETSC ERROR: #6 DMGlobalToLocalEnd() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/dm/interface/dm.c:2849
[5]PETSC ERROR: #7 SNESComputeFunction_DMLocal() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/utils/dmlocalsnes.c:65
[5]PETSC ERROR: #8 SNES callback function
[5]PETSC ERROR: #9 SNESComputeFunction() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:2436
[5]PETSC ERROR: #10 SNESSolve_KSPONLY() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ksponly/ksponly.c:27
[5]PETSC ERROR: #11 SNESSolve() at
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4690
[5]PETSC ERROR: #12 main() at ex13.c:178
MPICH ERROR [Rank 5] [job id 213404.0] [Wed Nov 16 18:30:49 2022]
[crusher011] - Abort(59) (rank 5 in comm 0): application called
MPI_Abort(MPI_COMM_WORLD, 59) - process 5

On Wed, Nov 16, 2022 at 2:51 PM Mark Adams <mfadams at lbl.gov> wrote:

> I am able to reproduce this on Crusher with 8 processors.
>
> Junchao, did you want me to use --download-kokkos-commit=origin/develop ?
>
> On Wed, Nov 16, 2022 at 8:05 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> I can not build right now on Crusher or Perlmutter but I saw this on both.
>>
>> Here is an example output using src/snes/tests/ex13.c using the appended
>> .petscrc
>> This uses 64 processors and the 8 processor case worked. This has been
>> semi-nondertminisitc for me.
>>
>> (and I have attached my current Perlmutter problem)
>>
>> Hope this helps,
>> Mark
>>
>> -dm_plex_simplex 0
>> -dm_plex_dim 3
>> -dm_plex_box_lower 0,0,0
>> -dm_plex_box_upper 1,1,1
>> -petscpartitioner_simple_process_grid 2,2,2
>> -potential_petscspace_degree 2
>> -snes_max_it 1
>> -ksp_max_it 200
>> -ksp_type cg
>> -ksp_rtol 1.e-12
>> -ksp_norm_type unpreconditioned
>> -snes_rtol 1.e-8
>> #-pc_type gamg
>> #-pc_gamg_type agg
>> #-pc_gamg_agg_nsmooths 1
>> -pc_gamg_coarse_eq_limit 100
>> -pc_gamg_process_eq_limit 400
>> -pc_gamg_reuse_interpolation true
>> #-snes_monitor
>> #-ksp_monitor_short
>> -ksp_converged_reason
>> #-ksp_view
>> #-snes_converged_reason
>> #-mg_levels_ksp_max_it 2
>> -mg_levels_ksp_type chebyshev
>> #-mg_levels_ksp_type richardson
>> #-mg_levels_ksp_richardson_scale 0.8
>> -mg_levels_pc_type jacobi
>> -pc_gamg_esteig_ksp_type cg
>> -pc_gamg_esteig_ksp_max_it 10
>> -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05
>> -dm_distribute
>> -petscpartitioner_type simple
>> -pc_gamg_repartition false
>> -pc_gamg_coarse_grid_layout_type compact
>> -pc_gamg_threshold 0.01
>> #-pc_gamg_threshold_scale .5
>> -pc_gamg_aggressive_coarsening 1
>> #-check_pointer_intensity 0
>> -snes_type ksponly
>> #-mg_coarse_sub_pc_factor_mat_solver_type cusparse
>> #-info :pc
>> #-use_gpu_aware_mpi 1
>> -options_left
>> #-malloc_debug
>> -benchmark_it 10
>> #-pc_gamg_use_parallel_coarse_grid_solver
>> #-mg_coarse_pc_type jacobi
>> #-mg_coarse_ksp_type cg
>> #-mg_coarse_ksp_rtol 1.e-2
>> #-mat_cusparse_transgen
>> -snes_lag_jacobian -2
>>
>>
>> On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <junchao.zhang at gmail.com>
>> wrote:
>>
>>> Mark,
>>> Do you have a reproducer using petsc examples?
>>>
>>> On Tue, Nov 15, 2022, 12:49 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> Junchao, this is the same problem that I have been having right?
>>>>
>>>> On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> I built petsc with:
>>>>>
>>>>> $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug
>>>>> --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0
>>>>> --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices
>>>>> --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos
>>>>> --download-kokkos-kernels
>>>>>
>>>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
>>>>>
>>>>> $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
>>>>>
>>>>>
>>>>> Then I build xolotl in a separate build directory (after checking out
>>>>> the "feature-petsc-kokkos" branch) with:
>>>>>
>>>>> $ cmake -DCMAKE_BUILD_TYPE=Debug
>>>>> -DKokkos_DIR=$HOME/build/petsc/debug/install
>>>>> -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
>>>>>
>>>>> $ make -j4 SystemTester
>>>>>
>>>>>
>>>>> Then, from the xolotl build directory, run (for example):
>>>>>
>>>>> $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
>>>>>
>>>>> Note that this test case will use the parameter file
>>>>> '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line
>>>>> arguments for petsc in its "petscArgs=..." line. If you look at
>>>>> '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases
>>>>> follow the same naming convention with their corresponding parameter files
>>>>> under '<xolotl-src>/benchmarks'.
>>>>>
>>>>> The failure happens with the NE_4 case (which is 2D) and the PSI_3
>>>>> case (which is 1D).
>>>>>
>>>>> Let me know if this is still unclear.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> *Philip Fackler *
>>>>> Research Software Engineer, Application Engineering Group
>>>>> Advanced Computing Systems Research Section
>>>>> Computer Science and Mathematics Division
>>>>> *Oak Ridge National Laboratory*
>>>>> ------------------------------
>>>>> *From:* Junchao Zhang <junchao.zhang at gmail.com>
>>>>> *Sent:* Tuesday, November 15, 2022 00:16
>>>>> *To:* Fackler, Philip <facklerpw at ornl.gov>
>>>>> *Cc:* petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>; Blondel,
>>>>> Sophie <sblondel at utk.edu>
>>>>> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with
>>>>> COO interface crashes in some cases
>>>>>
>>>>> Hi, Philip,
>>>>>   Can you tell me instructions to build Xolotl to reproduce the error?
>>>>> --Junchao Zhang
>>>>>
>>>>>
>>>>> On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <
>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>
>>>>> In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to
>>>>> use the COO interface for preallocating and setting values in the Jacobian
>>>>> matrix. I have found that with some of our test cases, using more than one
>>>>> MPI rank results in a crash. Way down in the preconditioner code in petsc a
>>>>> Mat gets computed that has "null" for the "productsymbolic" member of its
>>>>> "ops". It's pretty far removed from where we compute the Jacobian entries,
>>>>> so I haven't been able (so far) to track it back to an error in my code.
>>>>> I'd appreciate some help with this from someone who is more familiar with
>>>>> the petsc guts so we can figure out what I'm doing wrong. (I'm assuming
>>>>> it's a bug in Xolotl.)
>>>>>
>>>>> Note that this is using the kokkos backend for Mat and Vec in petsc,
>>>>> but with a serial-only build of kokkos and kokkos-kernels. So, it's a
>>>>> CPU-only multiple MPI rank run.
>>>>>
>>>>> Here's a paste of the error output showing the relevant parts of the
>>>>> call stack:
>>>>>
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] --------------------- Error Message
>>>>> --------------------------------------------------------------
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] No support for this operation for this object type
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] No support for this operation for this object type
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] No method productsymbolic for Mat of type (null)
>>>>> [ERROR] No method productsymbolic for Mat of type (null)
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting.
>>>>> [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting.
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT
>>>>> Date: 2022-10-28 14:39:41 +0000
>>>>> [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT
>>>>> Date: 2022-10-28 14:39:41 +0000
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01
>>>>> 2022
>>>>> [ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01
>>>>> 2022
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
>>>>> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
>>>>> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
>>>>> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
>>>>> --with-shared-libraries
>>>>> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
>>>>> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
>>>>> [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc
>>>>> PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc
>>>>> --with-cxx=mpicxx --with-fc=0 --with-cudac=0
>>>>> --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices
>>>>> --with-shared-libraries
>>>>> --with-kokkos-dir=/home/4pf/build/kokkos/serial/install
>>>>> --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
>>>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
>>>>> [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at
>>>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
>>>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
>>>>> [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at
>>>>> /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #3 MatProductSymbolic() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
>>>>> [ERROR] #3 MatProductSymbolic() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #4 MatProduct_Private() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
>>>>> [ERROR] #4 MatProduct_Private() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #5 MatMatMult() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
>>>>> [ERROR] #5 MatMatMult() at
>>>>> /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #6 PCGAMGOptProlongator_AGG() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
>>>>> [ERROR] #6 PCGAMGOptProlongator_AGG() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #7 PCSetUp_GAMG() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
>>>>> [ERROR] #7 PCSetUp_GAMG() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #8 PCSetUp() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
>>>>> [ERROR] #8 PCSetUp() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #9 KSPSetUp() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
>>>>> [ERROR] #9 KSPSetUp() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #10 KSPSolve_Private() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
>>>>> [ERROR] #10 KSPSolve_Private() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #11 KSPSolve() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>>>> [ERROR] #11 KSPSolve() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #12 PCApply_FieldSplit() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
>>>>> [ERROR] #12 PCApply_FieldSplit() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #13 PCApply() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
>>>>> [ERROR] #13 PCApply() at
>>>>> /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #14 KSP_PCApply() at
>>>>> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
>>>>> [ERROR] #14 KSP_PCApply() at
>>>>> /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #15 KSPFGMRESCycle() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
>>>>> [ERROR] #15 KSPFGMRESCycle() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #16 KSPSolve_FGMRES() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
>>>>> [ERROR] #16 KSPSolve_FGMRES() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #17 KSPSolve_Private() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
>>>>> [ERROR] #17 KSPSolve_Private() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #18 KSPSolve() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>>>> [ERROR] #18 KSPSolve() at
>>>>> /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] #19 SNESSolve_NEWTONLS() at
>>>>> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
>>>>> [ERROR] #19 SNESSolve_NEWTONLS() at
>>>>> /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #20 SNESSolve() at
>>>>> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
>>>>> [ERROR] #20 SNESSolve() at
>>>>> /home/4pf/repos/petsc/src/snes/interface/snes.c:4689
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #21 TSStep_ARKIMEX() at
>>>>> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
>>>>> [ERROR] #21 TSStep_ARKIMEX() at
>>>>> /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #22 TSStep() at
>>>>> /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
>>>>> [ERROR] #22 TSStep() at
>>>>> /home/4pf/repos/petsc/src/ts/interface/ts.c:3445
>>>>> [ERROR] [1]PETSC ERROR:
>>>>> [ERROR] [0]PETSC ERROR:
>>>>> [ERROR] #23 TSSolve() at
>>>>> /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
>>>>> [ERROR] #23 TSSolve() at
>>>>> /home/4pf/repos/petsc/src/ts/interface/ts.c:3836
>>>>> [ERROR] PetscSolver::solve: TSSolve failed.
>>>>> [ERROR] PetscSolver::solve: TSSolve failed.
>>>>> Aborting.
>>>>> Aborting.
>>>>>
>>>>>
>>>>>
>>>>> Thanks for the help,
>>>>>
>>>>>
>>>>> *Philip Fackler *
>>>>> Research Software Engineer, Application Engineering Group
>>>>> Advanced Computing Systems Research Section
>>>>> Computer Science and Mathematics Division
>>>>> *Oak Ridge National Laboratory*
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221116/525242b0/attachment-0001.html>