(GTL DEBUG: 12) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: General MPI error 
[12]PETSC ERROR: MPI error 1 Invalid buffer pointer
[12]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[12]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[12]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002201 by sajid Thu Feb 10 15:49:31 2022
[12]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[12]PETSC ERROR: #1 PetscSFLinkWaitRequests_MPI() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfmpi.c:54
[12]PETSC ERROR: #2 PetscSFLinkFinishCommunication() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/include/../src/vec/is/sf/impls/basic/sfpack.h:274
[12]PETSC ERROR: #3 PetscSFReduceEnd_Basic() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/impls/basic/sfbasic.c:259
[12]PETSC ERROR: #4 PetscSFReduceEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1601
[12]PETSC ERROR: #5 PetscSFGatherEnd() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/vec/is/sf/interface/sf.c:1896
[12]PETSC ERROR: #6 MatProductNumeric_MPIAIJBACKEND() at /tmp/sajid/spack-stage/spack-stage-petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo/spack-src/src/mat/impls/aij/mpi/mpiaij.c:6491
MPICH Notice [Rank 12] [job id 1407674.0] [Thu Feb 10 15:49:53 2022] [nid002201] - Abort(98) (rank 12 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 98) - process 12

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 98) - process 12
Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
(GTL DEBUG: 15) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
(GTL DEBUG: 14) cuIpcOpenMemHandle: resource already mapped, CUDA_ERROR_ALREADY_MAPPED, line no 272
srun: error: nid002201: task 12: Exited with exit code 255
srun: launch/slurm: _step_signal: Terminating StepId=1407674.0
slurmstepd: error: *** STEP 1407674.0 ON nid002196 CANCELLED AT 2022-02-10T23:49:54 ***
[28]PETSC ERROR: ------------------------------------------------------------------------
[28]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[28]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[28]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[28]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[28]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[28]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[28]PETSC ERROR: to get more information on the crash.
[29]PETSC ERROR: ------------------------------------------------------------------------
[29]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[29]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[29]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[29]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[29]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[29]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[29]PETSC ERROR: to get more information on the crash.
[29]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[29]PETSC ERROR: Signal received
[29]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[29]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[29]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002209 by sajid Thu Feb 10 15:49:31 2022
[29]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[30]PETSC ERROR: ------------------------------------------------------------------------
[30]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[30]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[30]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[30]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[30]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[30]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[30]PETSC ERROR: to get more information on the crash.
[30]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[30]PETSC ERROR: Signal received
[30]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[30]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[30]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002209 by sajid Thu Feb 10 15:49:31 2022
[30]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[31]PETSC ERROR: ------------------------------------------------------------------------
[31]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[31]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[31]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[31]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[31]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[31]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[31]PETSC ERROR: to get more information on the crash.
[31]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[31]PETSC ERROR: Signal received
[31]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[31]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[31]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002209 by sajid Thu Feb 10 15:49:31 2022
[31]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[28]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[28]PETSC ERROR: Signal received
[28]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[28]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[28]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002209 by sajid Thu Feb 10 15:49:31 2022
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[28]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[1]PETSC ERROR: to get more information on the crash.
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[3]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[3]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[3]PETSC ERROR: to get more information on the crash.
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Signal received
[3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[3]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[3]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002196 by sajid Thu Feb 10 15:49:31 2022
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[0]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002196 by sajid Thu Feb 10 15:49:31 2022
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[0]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kerln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[29]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 29] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002209] - Abort(59) (rank 29 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 29

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 29
[24]PETSC ERROR: ------------------------------------------------------------------------
[24]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[24]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[24]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[24]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[24]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[24]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[24]PETSC ERROR: to get more information on the crash.
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctlrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[1]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002196 by sajid Thu Feb 10 15:49:31 2022
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[30]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 30] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002209] - Abort(59) (rank 30 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 30

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 30
[1]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[25]PETSC ERROR: ------------------------------------------------------------------------
[25]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[25]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[25]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[25]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[25]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[25]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[25]PETSC ERROR: to get more information on the crash.
[25]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[25]PETSC ERROR: Signal received
[25]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[25]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[25]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002208 by sajid Thu Feb 10 15:49:31 2022
[14]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[2]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[2]PETSC ERROR: to get more information on the crash.
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[2]PETSC ERROR: Signal received
[2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[26]PETSC ERROR: ------------------------------------------------------------------------
[26]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[26]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[26]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[26]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[26]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[26]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[26]PETSC ERROR: to get more information on the crash.
[26]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[26]PETSC ERROR: Signal received
[26]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[2]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[2]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002196 by sajid Thu Feb 10 15:49:31 2022
[26]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[26]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002208 by sajid Thu Feb 10 15:49:31 2022
[2]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/[26]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[31]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 31] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002209] - Abort(59) (rank 31 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 31

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 31
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[28]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 28] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002209] - Abort(59) (rank 28 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28
[27]PETSC ERROR: ------------------------------------------------------------------------
[27]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[27]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[27]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[27]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[27]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[27]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[27]PETSC ERROR: to get more information on the crash.
[27]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[27]PETSC ERROR: Signal received
[27]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[27]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[27]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002208 by sajid Thu Feb 10 15:49:31 2022
[27]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[24]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[24]PETSC ERROR: Signal received
[24]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[24]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[24]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002208 by sajid Thu Feb 10 15:49:31 2022
[24]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[14]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[14]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[14]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[14]PETSC ERROR: to get more information on the crash.
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[25]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[15]PETSC ERROR: ------------------------------------------------------------------------
[15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[15]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[13]PETSC ERROR: ------------------------------------------------------------------------
[13]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[13]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[13]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[13]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[13]PETSC ERROR: to get more information on the crash.
[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[14]PETSC ERROR: Signal received
[14]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[14]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[14]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002201 by sajid Thu Feb 10 15:49:31 2022
[15]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[15]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[15]PETSC ERROR: to get more information on the crash.
[15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[15]PETSC ERROR: Signal received
[15]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[15]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[15]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002201 by sajid Thu Feb 10 15:49:31 2022
[8]PETSC ERROR: ------------------------------------------------------------------------
[8]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[8]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[8]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[8]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[8]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[25]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 25] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002208] - Abort(59) (rank 25 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 25

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 25
[9]PETSC ERROR: ------------------------------------------------------------------------
[9]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[9]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[9]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[9]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[9]PETSC ERROR: to get more information on the crash.
[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[9]PETSC ERROR: Signal received
[9]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[13]PETSC ERROR: Signal received
[13]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[13]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[13]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002201 by sajid Thu Feb 10 15:49:31 2022
[9]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[9]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002200 by sajid Thu Feb 10 15:49:31 2022
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[26]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 26] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002208] - Abort(59) (rank 26 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 26

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 26
[14]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[27]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 27] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002208] - Abort(59) (rank 27 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 27

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 27
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[24]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 24] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002208] - Abort(59) (rank 24 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 24

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 24
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[14]PETSC ERROR: #1 User provided function() at unknown file:0
[15]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[4]PETSC ERROR: to get more information on the crash.
[10]PETSC ERROR: ------------------------------------------------------------------------
[10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[10]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[10]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[10]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[10]PETSC ERROR: to get more information on the crash.
[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[10]PETSC ERROR: Signal received
[10]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[16]PETSC ERROR: ------------------------------------------------------------------------
[16]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[16]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[16]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[16]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[16]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[16]PETSC ERROR: to get more information on the crash.
[10]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[10]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002200 by sajid Thu Feb 10 15:49:31 2022
[17]PETSC ERROR: ------------------------------------------------------------------------
[17]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[17]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[17]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[17]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[17]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[17]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[17]PETSC ERROR: to get more information on the crash.
[17]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[17]PETSC ERROR: Signal received
[17]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[11]PETSC ERROR: ------------------------------------------------------------------------
[11]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[11]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[11]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[11]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[11]PETSC ERROR: to get more information on the crash.
[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[11]PETSC ERROR: Signal received
[11]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[5]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[5]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[5]PETSC ERROR: to get more information on the crash.
[17]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[17]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002204 by sajid Thu Feb 10 15:49:31 2022
[11]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[11]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002200 by sajid Thu Feb 10 15:49:31 2022
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[6]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[6]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[6]PETSC ERROR: to get more information on the crash.
[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[6]PETSC ERROR: Signal received
[6]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[18]PETSC ERROR: ------------------------------------------------------------------------
[18]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[18]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[18]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[18]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[18]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[18]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[18]PETSC ERROR: to get more information on the crash.
[18]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[18]PETSC ERROR: Signal received
[18]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[11]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[6]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[6]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002197 by sajid Thu Feb 10 15:49:31 2022
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[18]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[18]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002204 by sajid Thu Feb 10 15:49:31 2022
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[7]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[7]PETSC ERROR: to get more information on the crash.
[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[7]PETSC ERROR: Signal received
[7]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[15]PETSC ERROR: #1 User provided function() at unknown file:0
[8]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[8]PETSC ERROR: to get more information on the crash.
[8]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[8]PETSC ERROR: Signal received
[8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[8]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[8]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002200 by sajid Thu Feb 10 15:49:31 2022
[13]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[8]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt//spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kespack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kerrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctlln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[13]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 13] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002201] - Abort(59) (rank 13 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 13
MPICH Notice [Rank 14] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002201] - Abort(59) (rank 14 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 14
[20]PETSC ERROR: ------------------------------------------------------------------------
[20]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[20]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[20]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[20]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[20]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[20]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[20]PETSC ERROR: to get more information on the crash.
[7]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[7]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002197 by sajid Thu Feb 10 15:49:31 2022
[9]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/[7]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/[18]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kerspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[21]PETSC ERROR: ------------------------------------------------------------------------
[21]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[21]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[21]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[21]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[21]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[21]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[21]PETSC ERROR: to get more information on the crash.
[21]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[21]PETSC ERROR: Signal received
[21]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[4]PETSC ERROR: Signal received
[4]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[4]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[4]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002197 by sajid Thu Feb 10 15:49:31 2022
[19]PETSC ERROR: ------------------------------------------------------------------------
[19]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[19]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[19]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[19]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[19]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[19]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[19]PETSC ERROR: to get more information on the crash.
[19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: Signal received
[19]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[21]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[21]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002205 by sajid Thu Feb 10 15:49:31 2022
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[9]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 9] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002200] - Abort(59) (rank 9 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 9
[4]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/[19]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[19]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002204 by sajid Thu Feb 10 15:49:31 2022
[22]PETSC ERROR: ------------------------------------------------------------------------
[22]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[22]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[22]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[22]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[22]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[22]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[22]PETSC ERROR: to get more information on the crash.
[22]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[22]PETSC ERROR: Signal received
[22]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[10]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optspack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[19]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[22]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[22]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002205 by sajid Thu Feb 10 15:49:31 2022
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keMPICH Notice [Rank 15] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002201] - Abort(59) (rank 15 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 15
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Signal received
[16]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[16]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[16]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002204 by sajid Thu Feb 10 15:49:31 2022
[16]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[22]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[23]PETSC ERROR: ------------------------------------------------------------------------
[23]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
[23]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[23]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[23]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[23]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems  to find memory corruption errors
[23]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[23]PETSC ERROR: to get more information on the crash.
[23]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[23]PETSC ERROR: Signal received
[23]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[23]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[23]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002205 by sajid Thu Feb 10 15:49:31 2022
[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[5]PETSC ERROR: Signal received
[5]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[5]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[5]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002197 by sajid Thu Feb 10 15:49:31 2022
[23]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[5]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl[6]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctl/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[17]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt[20]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[20]PETSC ERROR: Signal received
[20]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[20]PETSC ERROR: Petsc Development GIT revision: f351d5494b5462f62c419e00645ac2e477b88cae  GIT Date: 2022-02-08 15:08:19 +0000
[20]PETSC ERROR: /global/homes/s/sajid/packages/aclatfd/3D/poisson3d on a  named nid002205 by sajid Thu Feb 10 15:49:31 2022
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[10]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 10] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002200] - Abort(59) (rank 10 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ke[20]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-keln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[11]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 11] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002200] - Abort(59) (rank 11 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 11
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[17]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 17] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002204] - Abort(59) (rank 17 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 17
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctn7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[8]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 8] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002200] - Abort(59) (rank 8 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 8
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtct[21]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/optn7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[6]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 6] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002197] - Abort(59) (rank 6 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[18]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 18] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002204] - Abort(59) (rank 18 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 18
/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kenels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctlrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctrnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctn7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[7]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 7] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002197] - Abort(59) (rank 7 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 7
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[19]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 19] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002204] - Abort(59) (rank 19 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 19
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[21]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 21] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002205] - Abort(59) (rank 21 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[16]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 16] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002204] - Abort(59) (rank 16 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 16
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[22]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 22] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002205] - Abort(59) (rank 22 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 22
rnels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[23]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 23] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002205] - Abort(59) (rank 23 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 23
ln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[20]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 20] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002205] - Abort(59) (rank 20 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 20
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[4]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 4] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002197] - Abort(59) (rank 4 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[5]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 5] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002197] - Abort(59) (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5
spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-ker[3]PETSC ERROR: Configure options --prefix=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/petsc-main-mnj56kbexro3fipf6kheyttljzwss7fo --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 CFLAGS= FFLAGS= CXXFLAGS= --with-cc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicc --with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpicxx --with-fc=/opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=0 --with-openmp=0 --with-64-bit-indices=0 COPTFLAGS= FOPTFLAGS= CXXOPTFLAGS= --with-blaslapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4 --with-hip=0 --with-metis=1 --with-metis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/include --with-metis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/lib/libmetis.so --with-hypre=1 --with-hypre-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/include --with-hypre-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hypre-develop-3gtrobj6ky64qlq4jvi2qzou5mvisy4w/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/include --with-parmetis-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/parmetis-4.0.3-7xhbi6h22ni4fe35vxurnwmr6izbeb7b/lib/libparmetis.so --with-kokkos=1 --with-kokkos-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-3.5.00-65sqphcwz6lwtqectq6yswa6kt3654mb --with-kokkos-kernels=1 --with-kokkos-kernels-dir=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/kokkos-kernels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[3]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 3] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002196] - Abort(59) (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 3
n7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[0]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 0] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002196] - Abort(59) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
[1]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 1] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002196] - Abort(59) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
nels-3.5.00-zwq3aedpbg7ywpmqiqxmn5nx4w6hdrx6 --with-superlu_dist=1 --with-superlu_dist-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/include --with-superlu_dist-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/superlu-dist-develop-l5kc2sttvfqcjlejhgnvygfxwulrujga/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/include --with-hdf5-lib="/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5_hl.so /global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.12.1-7pefaoio5q3hwnzggbnz7mpqw352gtsy/lib/libhdf5.so" --with-zlib=1 --with-zlib-include=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/include --with-zlib-lib=/global/u1/s/sajid/packages/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.11-ekeupmdcqoimgroigtctln7tqkyh6pdm/lib/libz.so --with-mumps=0 --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0 --with-cuda-arch=80
[2]PETSC ERROR: #1 User provided function() at unknown file:0
MPICH Notice [Rank 2] [job id 1407674.0] [Thu Feb 10 15:49:54 2022] [nid002196] - Abort(59) (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2
srun: error: nid002209: task 29: Exited with exit code 15
srun: error: nid002196: task 1: Exited with exit code 15
srun: error: nid002201: tasks 13,15: Exited with exit code 15
srun: error: nid002204: task 16: Exited with exit code 15
srun: error: nid002200: task 9: Exited with exit code 15
srun: error: nid002205: task 22: Exited with exit code 15
srun: error: nid002208: task 27: Exited with exit code 15
srun: error: nid002197: task 6: Exited with exit code 15
srun: error: nid002196: tasks 0,3: Exited with exit code 15
srun: error: nid002209: tasks 30-31: Exited with exit code 15
srun: error: nid002204: tasks 18-19: Exited with exit code 15
srun: error: nid002197: tasks 5,7: Exited with exit code 15
srun: error: nid002205: tasks 20-21: Exited with exit code 15
srun: error: nid002201: task 14: Exited with exit code 15
srun: error: nid002208: tasks 25-26: Exited with exit code 15
srun: error: nid002200: tasks 8,11: Exited with exit code 15
srun: error: nid002196: task 2: Exited with exit code 15
srun: error: nid002209: task 28: Exited with exit code 15
srun: error: nid002204: task 17: Exited with exit code 15
srun: error: nid002205: task 23: Exited with exit code 15
srun: error: nid002208: task 24: Exited with exit code 15
srun: error: nid002197: task 4: Exited with exit code 15
srun: error: nid002200: task 10: Exited with exit code 15