<div dir="ltr">I can not build right now on Crusher or Perlmutter but I saw this on both.<div><br><div>Here is an example output using src/snes/tests/ex13.c using the appended .petscrc</div><div>This uses 64 processors and the 8 processor case worked. This has been semi-nondertminisitc for me.</div><div><br></div><div>(and I have attached my current Perlmutter problem) </div><div><br></div><div>Hope this helps,</div><div>Mark</div><div><br></div><div>-dm_plex_simplex 0<br>-dm_plex_dim 3<br>-dm_plex_box_lower 0,0,0<br>-dm_plex_box_upper 1,1,1<br>-petscpartitioner_simple_process_grid 2,2,2<br>-potential_petscspace_degree 2<br>-snes_max_it 1<br>-ksp_max_it 200<br>-ksp_type cg<br>-ksp_rtol 1.e-12<br>-ksp_norm_type unpreconditioned<br>-snes_rtol 1.e-8<br>#-pc_type gamg<br>#-pc_gamg_type agg<br>#-pc_gamg_agg_nsmooths 1<br>-pc_gamg_coarse_eq_limit 100<br>-pc_gamg_process_eq_limit 400<br>-pc_gamg_reuse_interpolation true<br>#-snes_monitor<br>#-ksp_monitor_short<br>-ksp_converged_reason<br>#-ksp_view<br>#-snes_converged_reason<br>#-mg_levels_ksp_max_it 2<br>-mg_levels_ksp_type chebyshev<br>#-mg_levels_ksp_type richardson<br>#-mg_levels_ksp_richardson_scale 0.8<br>-mg_levels_pc_type jacobi<br>-pc_gamg_esteig_ksp_type cg<br>-pc_gamg_esteig_ksp_max_it 10<br>-mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05<br>-dm_distribute<br>-petscpartitioner_type simple<br>-pc_gamg_repartition false<br>-pc_gamg_coarse_grid_layout_type compact<br>-pc_gamg_threshold 0.01<br>#-pc_gamg_threshold_scale .5<br>-pc_gamg_aggressive_coarsening 1<br>#-check_pointer_intensity 0<br>-snes_type ksponly<br>#-mg_coarse_sub_pc_factor_mat_solver_type cusparse<br>#-info :pc<br>#-use_gpu_aware_mpi 1<br>-options_left<br>#-malloc_debug<br>-benchmark_it 10<br>#-pc_gamg_use_parallel_coarse_grid_solver<br>#-mg_coarse_pc_type jacobi<br>#-mg_coarse_ksp_type cg<br>#-mg_coarse_ksp_rtol 1.e-2<br>#-mat_cusparse_transgen<br>-snes_lag_jacobian -2<br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Mark, <div dir="auto">Do you have a reproducer using petsc examples?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 15, 2022, 12:49 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span style="color:rgb(80,0,80)">Junchao, this is the same problem that I have been having right?</span><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>




<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
I built petsc with:</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos
 --download-kokkos-kernels</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src><br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
$ make -j4 SystemTester</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<u><i><b><br>
</b></i></u></div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Then, from the xolotl build directory, run (for example):</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test
 cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
Let me know if this is still unclear.<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
Thanks,<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_5961006331001504245m_-3703806045038936495m_-2327645382633610414Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
<div id="m_5961006331001504245m_-3703806045038936495m_-2327645382633610414appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_5961006331001504245m_-3703806045038936495m_-2327645382633610414divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" rel="noreferrer" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Tuesday, November 15, 2022 00:16<br>
<b>To:</b> Fackler, Philip <<a href="mailto:facklerpw@ornl.gov" rel="noreferrer" target="_blank">facklerpw@ornl.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer" target="_blank">petsc-users@mcs.anl.gov</a>>; Blondel, Sophie <<a href="mailto:sblondel@utk.edu" rel="noreferrer" target="_blank">sblondel@utk.edu</a>><br>
<b>Subject:</b> [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi, Philip,
<div>  <span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Can you tell me instructions to build Xolotl to reproduce the error?</span><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" rel="noreferrer" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in
 the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code.
 I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Here's a paste of the error output showing the relevant parts of the call stack:</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
[ERROR] [0]PETSC ERROR:
<div>[ERROR] --------------------- Error Message --------------------------------------------------------------</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] --------------------- Error Message --------------------------------------------------------------</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] No support for this operation for this object type</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] No support for this operation for this object type</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] No method productsymbolic for Mat of type (null)</div>
<div>[ERROR] No method productsymbolic for Mat of type (null)</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] See <a rel="noreferrer">hxxps://petsc.org/release/faq/</a> for trouble shooting.</div>
<div>[ERROR] See <a rel="noreferrer">hxxps://petsc.org/release/faq/</a> for trouble shooting.</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT Date: 2022-10-28 14:39:41 +0000</div>
<div>[ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a  GIT Date: 2022-10-28 14:39:41 +0000</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01 2022</div>
<div>[ERROR] Unknown Name on a  named PC0115427 by 4pf Mon Nov 14 13:22:01 2022</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries
 --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install</div>
<div>[ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries
 --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918</div>
<div>[ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138</div>
<div>[ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793</div>
<div>[ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820</div>
<div>[ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897</div>
<div>[ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769</div>
<div>[ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639</div>
<div>[ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994</div>
<div>[ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406</div>
<div>[ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825</div>
<div>[ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div>[ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246</div>
<div>[ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441</div>
<div>[ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380</div>
<div>[ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152</div>
<div>[ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273</div>
<div>[ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899</div>
<div>[ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div>[ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071</div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210</div>
<div>[ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689</div>
<div>[ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791</div>
<div>[ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445</div>
<div>[ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445</div>
<div>[ERROR] [1]PETSC ERROR: </div>
<div>[ERROR] [0]PETSC ERROR: </div>
<div>[ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836</div>
<div>[ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836</div>
<div>[ERROR] PetscSolver::solve: TSSolve failed.</div>
<div>[ERROR] PetscSolver::solve: TSSolve failed.</div>
<div>Aborting.</div>
Aborting.<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks for the help,<br>
</div>
<div>
<div style="font-family:Consolas,Courier,monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_5961006331001504245m_-3703806045038936495m_-2327645382633610414x_m_3242059876177914397m_3736712901589057600Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:11pt"><strong>Philip Fackler<br>
</strong></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div><span style="font-size:11pt">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-size:11pt">Advanced Computing Systems Research Section</span></div>
<div><span style="font-size:11pt">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-size:11pt"><strong>Oak Ridge National Laboratory</strong></span><span style="font-size:11pt"></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>

</div></blockquote></div>
</blockquote></div>
</blockquote></div>