(GTL DEBUG: 25) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
(GTL DEBUG: 24) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
(GTL DEBUG: 26) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: General MPI error 
[24]PETSC ERROR: MPI error 1 Invalid buffer pointer
[24]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[24]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[24]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003276 by sajid Wed Feb  9 20:06:32 2022
[24]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[24]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[24]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[24]PETSC ERROR: #3 PetscSFBcastEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:218
[24]PETSC ERROR: #4 PetscSFBcastEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1499
[24]PETSC ERROR: #5 VecScatterEnd_Internal() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:87
[24]PETSC ERROR: #6 VecScatterEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:1366
[24]PETSC ERROR: #7 MatMult_MPIAIJCUSPARSE() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:302
[24]PETSC ERROR: #8 MatMult() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/interface/matrix.c:2438
[24]PETSC ERROR: #9 PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:730
[24]PETSC ERROR: #10 KSP_PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/petsc/private/kspimpl.h:421
[24]PETSC ERROR: #11 KSPGMRESCycle() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:162
[24]PETSC ERROR: #12 KSPSolve_GMRES() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:247
[24]PETSC ERROR: #13 KSPSolve_Private() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:925
[24]PETSC ERROR: #14 KSPSolve() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:1103
[24]PETSC ERROR: #15 PCGAMGOptProlongator_AGG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/agg.c:1127
[24]PETSC ERROR: #16 PCSetUp_GAMG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/gamg.c:626
[24]PETSC ERROR: #17 PCSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:1017
[24]PETSC ERROR: #18 KSPSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:417
[24]PETSC ERROR: #19 main() at poisson3d.c:69
[24]PETSC ERROR: PETSc Option Table entries:
[24]PETSC ERROR: -dm_mat_type aijcusparse
[24]PETSC ERROR: -dm_vec_type cuda
[24]PETSC ERROR: -ksp_monitor
[24]PETSC ERROR: -ksp_norm_type unpreconditioned
[24]PETSC ERROR: -ksp_type cg
[24]PETSC ERROR: -ksp_view
[24]PETSC ERROR: -log_view
[24]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[24]PETSC ERROR: -mg_levels_ksp_type chebyshev
[24]PETSC ERROR: -mg_levels_pc_type jacobi
[24]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[24]PETSC ERROR: -pc_gamg_square_graph 1
[24]PETSC ERROR: -pc_gamg_threshold 0.0
[24]PETSC ERROR: -pc_gamg_threshold_scale 0.0
[24]PETSC ERROR: -pc_gamg_type agg
[24]PETSC ERROR: -pc_type gamg
[24]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
MPICH Notice [Rank 24] [job id 1405824.0] [Wed Feb  9 20:06:52 2022] [nid003276] - Abort(98) (rank 24 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 98) - process 24

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 98) - process 24
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[26]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: General MPI error 
[26]PETSC ERROR: MPI error 1 Invalid buffer pointer
[26]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[26]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[26]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003276 by sajid Wed Feb  9 20:06:32 2022
[26]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[26]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[26]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[26]PETSC ERROR: #3 PetscSFBcastEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:218
[26]PETSC ERROR: #4 PetscSFBcastEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1499
[26]PETSC ERROR: #5 VecScatterEnd_Internal() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:87
[26]PETSC ERROR: #6 VecScatterEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:1366
[26]PETSC ERROR: #7 MatMult_MPIAIJCUSPARSE() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:302
[26]PETSC ERROR: #8 MatMult() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/interface/matrix.c:2438
[26]PETSC ERROR: #9 PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:730
[26]PETSC ERROR: #10 KSP_PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/petsc/private/kspimpl.h:421
[26]PETSC ERROR: #11 KSPGMRESCycle() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:162
[26]PETSC ERROR: #12 KSPSolve_GMRES() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:247
[26]PETSC ERROR: #13 KSPSolve_Private() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:925
[26]PETSC ERROR: #14 KSPSolve() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:1103
[26]PETSC ERROR: #15 PCGAMGOptProlongator_AGG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/agg.c:1127
[26]PETSC ERROR: #16 PCSetUp_GAMG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/gamg.c:626
[26]PETSC ERROR: #17 PCSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:1017
[26]PETSC ERROR: #18 KSPSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:417
[26]PETSC ERROR: #19 main() at poisson3d.c:69
[26]PETSC ERROR: PETSc Option Table entries:
[26]PETSC ERROR: -dm_mat_type aijcusparse
[26]PETSC ERROR: -dm_vec_type cuda
[26]PETSC ERROR: -ksp_monitor
[26]PETSC ERROR: -ksp_norm_type unpreconditioned
[26]PETSC ERROR: -ksp_type cg
[26]PETSC ERROR: -ksp_view
[26]PETSC ERROR: -log_view
[26]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[26]PETSC ERROR: -mg_levels_ksp_type chebyshev
[26]PETSC ERROR: -mg_levels_pc_type jacobi
[26]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[26]PETSC ERROR: -pc_gamg_square_graph 1
[26]PETSC ERROR: -pc_gamg_threshold 0.0
[26]PETSC ERROR: -pc_gamg_threshold_scale 0.0
[26]PETSC ERROR: -pc_gamg_type agg
[26]PETSC ERROR: -pc_type gamg
[26]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
MPICH Notice [Rank 26] [job id 1405824.0] [Wed Feb  9 20:06:52 2022] [nid003276] - Abort(98) (rank 26 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 98) - process 26

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 98) - process 26
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
[25]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[25]PETSC ERROR: General MPI error 
[25]PETSC ERROR: MPI error 1 Invalid buffer pointer
[25]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[25]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[25]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003276 by sajid Wed Feb  9 20:06:32 2022
[25]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[25]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[25]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[25]PETSC ERROR: #3 PetscSFBcastEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:218
[25]PETSC ERROR: #4 PetscSFBcastEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1499
[25]PETSC ERROR: #5 VecScatterEnd_Internal() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:87
[25]PETSC ERROR: #6 VecScatterEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/vscat.c:1366
[25]PETSC ERROR: #7 MatMult_MPIAIJCUSPARSE() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:302
[25]PETSC ERROR: #8 MatMult() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/interface/matrix.c:2438
[25]PETSC ERROR: #9 PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:730
[25]PETSC ERROR: #10 KSP_PCApplyBAorAB() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/petsc/private/kspimpl.h:421
[25]PETSC ERROR: #11 KSPGMRESCycle() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:162
[25]PETSC ERROR: #12 KSPSolve_GMRES() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/impls/gmres/gmres.c:247
[25]PETSC ERROR: #13 KSPSolve_Private() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:925
[25]PETSC ERROR: #14 KSPSolve() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:1103
[25]PETSC ERROR: #15 PCGAMGOptProlongator_AGG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/agg.c:1127
[25]PETSC ERROR: #16 PCSetUp_GAMG() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/impls/gamg/gamg.c:626
[25]PETSC ERROR: #17 PCSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/pc/interface/precon.c:1017
[25]PETSC ERROR: #18 KSPSetUp() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/ksp/ksp/interface/itfunc.c:417
[25]PETSC ERROR: #19 main() at poisson3d.c:69
[25]PETSC ERROR: PETSc Option Table entries:
[25]PETSC ERROR: -dm_mat_type aijcusparse
[25]PETSC ERROR: -dm_vec_type cuda
[25]PETSC ERROR: -ksp_monitor
[25]PETSC ERROR: -ksp_norm_type unpreconditioned
[25]PETSC ERROR: -ksp_type cg
[25]PETSC ERROR: -ksp_view
[25]PETSC ERROR: -log_view
[25]PETSC ERROR: -mg_levels_esteig_ksp_type cg
[25]PETSC ERROR: -mg_levels_ksp_type chebyshev
[25]PETSC ERROR: -mg_levels_pc_type jacobi
[25]PETSC ERROR: -pc_gamg_agg_nsmooths 1
[25]PETSC ERROR: -pc_gamg_square_graph 1
[25]PETSC ERROR: -pc_gamg_threshold 0.0
[25]PETSC ERROR: -pc_gamg_threshold_scale 0.0
[25]PETSC ERROR: -pc_gamg_type agg
[25]PETSC ERROR: -pc_type gamg
[25]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------
MPICH Notice [Rank 25] [job id 1405824.0] [Wed Feb  9 20:06:52 2022] [nid003276] - Abort(98) (rank 25 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 98) - process 25

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 98) - process 25
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
srun: error: nid003276: tasks 24-26: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=1405824.0
[28]PETSC ERROR: ------------------------------------------------------------------------
[28]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[28]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[28]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[28]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[28]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[28]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[28]PETSC ERROR: to get more information on the crash.
[29]PETSC ERROR: ------------------------------------------------------------------------
[29]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[29]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[29]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[29]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[29]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[29]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[29]PETSC ERROR: to get more information on the crash.
[29]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[29]PETSC ERROR: Signal received
[29]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[30]PETSC ERROR: ------------------------------------------------------------------------
[30]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[30]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[30]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[30]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[30]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[30]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[30]PETSC ERROR: to get more information on the crash.
[30]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[30]PETSC ERROR: Signal received
[30]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[30]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[30]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003277 by sajid Wed Feb  9 20:06:32 2022
[31]PETSC ERROR: ------------------------------------------------------------------------
[31]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[31]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[31]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[31]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[31]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[31]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[31]PETSC ERROR: to get more information on the crash.
[31]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[31]PETSC ERROR: Signal received
[31]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
slurmstepd: error: *** STEP 1405824.0 ON nid002052 CANCELLED AT 2022-02-10T04:06:53 ***
[31]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[31]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003277 by sajid Wed Feb  9 20:06:32 2022
[28]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[28]PETSC ERROR: Signal received
[28]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[28]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[28]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003277 by sajid Wed Feb  9 20:06:32 2022
[28]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[29]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[29]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003277 by sajid Wed Feb  9 20:06:32 2022
[29]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[27]PETSC ERROR: ------------------------------------------------------------------------
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[29]PETSC ERROR: #1 User provided function() at unknown file:0
[30]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[30]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 30] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid003277] - Abort(59) (rank 30 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 30

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 30
[31]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[31]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 31] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid003277] - Abort(59) (rank 31 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 31

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 31
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[28]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 28] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid003277] - Abort(59) (rank 28 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28
MPICH Notice [Rank 29] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid003277] - Abort(59) (rank 29 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 29

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 29
[27]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[27]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[27]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[27]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[27]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[27]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[27]PETSC ERROR: to get more information on the crash.
[27]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[27]PETSC ERROR: Signal received
[27]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[27]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[27]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid003276 by sajid Wed Feb  9 20:06:32 2022
[27]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[27]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 27] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid003276] - Abort(59) (rank 27 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 27

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 27
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[1]PETSC ERROR: to get more information on the crash.
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[3]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[3]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[3]PETSC ERROR: to get more information on the crash.
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Signal received
[3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[0]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002052 by sajid Wed Feb  9 20:06:32 2022
[0]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[1]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002052 by sajid Wed Feb  9 20:06:32 2022
[1]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[2]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[2]PETSC ERROR: to get more information on the crash.
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Signal received
[2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[2]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002052 by sajid Wed Feb  9 20:06:32 2022
[2]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[3]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002052 by sajid Wed Feb  9 20:06:32 2022
[3]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[5]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[5]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[5]PETSC ERROR: to get more information on the crash.
[12]PETSC ERROR: ------------------------------------------------------------------------
[12]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[12]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[12]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[12]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[12]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[12]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[12]PETSC ERROR: to get more information on the crash.
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[6]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[6]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[6]PETSC ERROR: to get more information on the crash.
[4]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[4]PETSC ERROR: to get more information on the crash.
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[7]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[7]PETSC ERROR: to get more information on the crash.
[16]PETSC ERROR: ------------------------------------------------------------------------
[16]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[16]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[16]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[16]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[16]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[16]PETSC ERROR: to get more information on the crash.
[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[4]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[4]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002053 by sajid Wed Feb  9 20:06:32 2022
[13]PETSC ERROR: ------------------------------------------------------------------------
[13]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[13]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[13]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[13]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[13]PETSC ERROR: to get more information on the crash.
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Signal received
[5]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[5]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[5]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002053 by sajid Wed Feb  9 20:06:32 2022
[14]PETSC ERROR: ------------------------------------------------------------------------
[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[14]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[14]PETSC ERROR: to get more information on the crash.
[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Signal received
[14]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[20]PETSC ERROR: ------------------------------------------------------------------------
[20]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[20]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[20]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[20]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[20]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[20]PETSC ERROR: to get more information on the crash.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Signal received
[6]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[6]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[6]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002053 by sajid Wed Feb  9 20:06:32 2022
[14]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[14]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002913 by sajid Wed Feb  9 20:06:32 2022
[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Signal received
[7]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[7]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[7]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002053 by sajid Wed Feb  9 20:06:32 2022
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Signal received
[12]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[12]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[12]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002913 by sajid Wed Feb  9 20:06:32 2022
[5]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[12]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[21]PETSC ERROR: ------------------------------------------------------------------------
[21]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[21]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[21]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[21]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[21]PETSC ERROR: to get more information on the crash.
[21]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: Signal received
[21]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[5]PETSC ERROR: #1 User provided function() at unknown file:0
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[21]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[21]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002929 by sajid Wed Feb  9 20:06:32 2022
[6]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[22]PETSC ERROR: ------------------------------------------------------------------------
[22]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[22]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[22]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[22]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[22]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[22]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[22]PETSC ERROR: to get more information on the crash.
[22]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: Signal received
[22]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[22]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[22]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002929 by sajid Wed Feb  9 20:06:32 2022
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Signal received
[20]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[20]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[20]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002929 by sajid Wed Feb  9 20:06:32 2022
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[6]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 6] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002053] - Abort(59) (rank 6 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6
[20]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[4]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt//spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kespack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[17]PETSC ERROR: ------------------------------------------------------------------------
[17]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[17]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[17]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[17]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[17]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[17]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[17]PETSC ERROR: to get more information on the crash.
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[4]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 4] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002053] - Abort(59) (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4
MPICH Notice [Rank 5] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002053] - Abort(59) (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5
[13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Signal received
[13]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[13]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[13]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002913 by sajid Wed Feb  9 20:06:32 2022
[13]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[18]PETSC ERROR: ------------------------------------------------------------------------
[18]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[18]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[18]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[18]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[18]PETSC ERROR: to get more information on the crash.
[8]PETSC ERROR: ------------------------------------------------------------------------
[8]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[8]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[8]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[8]PETSC ERROR: to get more information on the crash.
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Signal received
[16]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[16]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[16]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002928 by sajid Wed Feb  9 20:06:32 2022
[14]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: Signal received
[17]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[17]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[17]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002928 by sajid Wed Feb  9 20:06:32 2022
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[14]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 14] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002913] - Abort(59) (rank 14 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14
[18]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: Signal received
[18]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[18]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[18]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002928 by sajid Wed Feb  9 20:06:32 2022
[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[15]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[15]PETSC ERROR: to get more information on the crash.
[15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Signal received
[15]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[15]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[15]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002913 by sajid Wed Feb  9 20:06:32 2022
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[21]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[19]PETSC ERROR: ------------------------------------------------------------------------
[19]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[19]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[19]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[19]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[19]PETSC ERROR: to get more information on the crash.
[19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: Signal received
[19]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[19]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[19]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002928 by sajid Wed Feb  9 20:06:32 2022
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[19]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[21]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 21] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002929] - Abort(59) (rank 21 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[22]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[15]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[16]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[12]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 12] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002913] - Abort(59) (rank 12 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 12

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 12
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[16]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 16] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002928] - Abort(59) (rank 16 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[7]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[22]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 22] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002929] - Abort(59) (rank 22 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22
spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[23]PETSC ERROR: ------------------------------------------------------------------------
[23]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[23]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[23]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[23]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[23]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[23]PETSC ERROR: to get more information on the crash.
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[23]PETSC ERROR: Signal received
[23]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[23]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[23]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002929 by sajid Wed Feb  9 20:06:32 2022
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[7]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 7] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002053] - Abort(59) (rank 7 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[13]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 13] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002913] - Abort(59) (rank 13 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13
[23]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[15]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 15] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002913] - Abort(59) (rank 15 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[20]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 20] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002929] - Abort(59) (rank 20 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[23]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 23] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002929] - Abort(59) (rank 23 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23
[17]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[9]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[9]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[9]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[9]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[9]PETSC ERROR: to get more information on the crash.
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[10]PETSC ERROR: ------------------------------------------------------------------------
[10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[10]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[10]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[10]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[10]PETSC ERROR: to get more information on the crash.
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[11]PETSC ERROR: ------------------------------------------------------------------------
[11]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[11]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[11]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[11]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[11]PETSC ERROR: to get more information on the crash.
[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[17]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 17] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002928] - Abort(59) (rank 17 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17
[8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Signal received
[8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[8]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[8]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002912 by sajid Wed Feb  9 20:06:32 2022
[18]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: Signal received
[9]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[9]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[9]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002912 by sajid Wed Feb  9 20:06:32 2022
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Signal received
[10]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[10]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[10]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002912 by sajid Wed Feb  9 20:06:32 2022
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[11]PETSC ERROR: Signal received
[11]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[11]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[11]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002912 by sajid Wed Feb  9 20:06:32 2022
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[18]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 18] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002928] - Abort(59) (rank 18 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18
[8]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[8]PETSC ERROR: #1 User provided function() at unknown file:0
[9]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[9]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 9] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002912] - Abort(59) (rank 9 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9
[10]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[19]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 19] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002928] - Abort(59) (rank 19 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[10]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 10] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002912] - Abort(59) (rank 10 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10
[11]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[11]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 11] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002912] - Abort(59) (rank 11 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11
MPICH Notice [Rank 8] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002912] - Abort(59) (rank 8 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8
spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[3]PETSC ERROR: #1 User provided function() at unknown file:0
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[0]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 0] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002052] - Abort(59) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[1]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 1] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002052] - Abort(59) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[2]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 2] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002052] - Abort(59) (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
MPICH Notice [Rank 3] [job id 1405824.0] [Wed Feb  9 20:06:53 2022] [nid002052] - Abort(59) (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
srun: error: nid003277: task 30: Exited with exit code 15
srun: error: nid002052: tasks 2-3: Exited with exit code 15
srun: error: nid002053: task 7: Exited with exit code 15
srun: error: nid003276: task 27: Exited with exit code 15
srun: error: nid002929: task 22: Exited with exit code 15
srun: error: nid002928: task 19: Exited with exit code 15
srun: error: nid002913: task 13: Exited with exit code 15
srun: error: nid002912: task 10: Exited with exit code 15
srun: error: nid003277: tasks 29,31: Exited with exit code 15
srun: error: nid002052: task 0: Exited with exit code 15
srun: error: nid002053: tasks 4,6: Exited with exit code 15
srun: error: nid002929: tasks 20-21: Exited with exit code 15
srun: error: nid002912: tasks 9,11: Exited with exit code 15
srun: error: nid002928: tasks 16-17: Exited with exit code 15
srun: error: nid002913: tasks 12,15: Exited with exit code 15
srun: error: nid003277: task 28: Exited with exit code 15
srun: error: nid002052: task 1: Exited with exit code 15
srun: error: nid002053: task 5: Exited with exit code 15
srun: error: nid002929: task 23: Exited with exit code 15
srun: error: nid002928: task 18: Exited with exit code 15
srun: error: nid002912: task 8: Exited with exit code 15
srun: error: nid002913: task 14: Exited with exit code 15