hello from 0 of 2 hello from 1 of 2 petsc initalized for 1 time(s) with errcode= 0 @ 0 petsc initalized for 1 time(s) with errcode= 0 @ 1 set mat for gpu with errcode= 0 @ 0 matsettype set mat for gpu with errcode= 0 @ 1 matsettype mat setup with errcode= 0 @ 0 mat ownership range with errcode= 0 @ 0 mat setup with errcode= 0 @ 1 mat ownership range with errcode= 0 @ 1 mat value set with errcode= 0 @ 0 mat value set with errcode= 0 @ 1 mat assumbly with errcode= 0 @ 0 mat assumbly with errcode= 0 @ 1 setup vec from mat with errcode= 0 @ 0 setup vec from mat with errcode= 0 @ 1 vec duplication with errcode= 0 @ 0 vec duplication with errcode= 0 @ 1 MatMult for rhs with errcode= 0 @ 0 MatMult for rhs with errcode= 0 @ 1 -------------------------------------------------------------------------- The call to cuEventCreate failed. This is a unrecoverable error and will cause the program to abort. Hostname: XPS-15 cuEventCreate return value: 801 Check the cuda.h file for what the return value means. -------------------------------------------------------------------------- -------------------------------------------------------------------------- The call to cuIpcGetEventHandle failed. This is a unrecoverable error and will cause the program to abort. cuIpcGetEventHandle return value: 1 Check the cuda.h file for what the return value means. -------------------------------------------------------------------------- -------------------------------------------------------------------------- The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol cannot be used. cuIpcGetMemHandle return value: 801 address: 0x500e43000 Check the cuda.h file for what the return value means. Perhaps a reboot of the node will clear the problem. -------------------------------------------------------------------------- KSP solve the system with errcode= 0 @ 0 KSP solve the system with errcode= 0 @ 1 AxPy for error estimation with errcode= 0 @ 0 AxPy for error estimation with errcode= 0 @ 1 run 1 Norm of error 0.6808E-03,iterations 25 KSP distroyed with errcode= 0 @ 1 KSP distroyed with errcode= 0 @ 0 vecs distroyed with errcode= 0 @ 1 vecs distroyed with errcode= 0 @ 0 mat distroyed with errcode= 0 @ 1 mat distroyed with errcode= 0 @ 0 PETSc finalized with errcode= 0 @ 1 PETSc finalized with errcode= 0 @ 0 petsc initalized for 2 time(s) with errcode= 0 @ 1 petsc initalized for 2 time(s) with errcode= 0 @ 0 set mat for gpu with errcode= 0 @ 1 matsettype set mat for gpu with errcode= 0 @ 0 matsettype mat setup with errcode= 0 @ 1 mat ownership range with errcode= 0 @ 1 mat setup with errcode= 0 @ 0 mat ownership range with errcode= 0 @ 0 mat value set with errcode= 0 @ 1 mat value set with errcode= 0 @ 0 mat assumbly with errcode= 0 @ 1 mat assumbly with errcode= 0 @ 0 setup vec from mat with errcode= 0 @ 1 setup vec from mat with errcode= 0 @ 0 vec duplication with errcode= 0 @ 1 vec duplication with errcode= 0 @ 0 MatMult for rhs with errcode= 0 @ 1 MatMult for rhs with errcode= 0 @ 0 KSP solve the system with errcode= 0 @ 1 AxPy for error estimation with errcode= 0 @ 1 KSP solve the system with errcode= 0 @ 0 AxPy for error estimation with errcode= 0 @ 0 run 2 Norm of error 0.6808E-03,iterations 25 KSP distroyed KSP distroyed with errcode= 0 @ 0 with errcode= 0 @ 1 vecs distroyed with errcode= 0 @ 0 vecs distroyed with errcode= 0 @ 1 mat distroyed mat distroyed with errcode= 0 @ 1 with errcode= 0 @ 0 PETSc finalized with errcode= 0 @ 1 PETSc finalized with errcode= 0 @ 0 petsc initalized for 3 time(s) with errcode= 0 @ 1 petsc initalized for 3 time(s) with errcode= 0 @ 0 set mat for gpu with errcode= 0 @ 1 matsettype set mat for gpu with errcode= 0 @ 0 matsettype mat setup with errcode= 0 @ 1 mat ownership range with errcode= 0 @ 1 mat setup with errcode= 0 @ 0 mat ownership range with errcode= 0 @ 0 mat value set with errcode= 0 @ 1 mat value set with errcode= 0 @ 0 mat assumbly with errcode= 0 @ 1 mat assumbly with errcode= 0 @ 0 setup vec from mat with errcode= 0 @ 0 setup vec from mat with errcode= 0 @ 1 vec duplication with errcode= 0 @ 0 vec duplication with errcode= 0 @ 1 MatMult for rhs with errcode= 0 @ 0 MatMult for rhs with errcode= 0 @ 1 KSP solve the system with errcode= 0 @ 0 AxPy for error estimation with errcode= 0 @ 0 KSP solve the system with errcode= 0 @ 1 AxPy for error estimation with errcode= 0 @ 1 run 3 Norm of error 0.6808E-03,iterations 25 KSP distroyed with errcode= 0 @ 1 KSP distroyed with errcode= 0 @ 0 vecs distroyed with errcode= 0 @ 1 vecs distroyed with errcode= 0 @ 0 mat distroyed with errcode= 0 @ 1 mat distroyed with errcode= 0 @ 0 PETSc finalized with errcode= 0 @ 1 goodbye from 1 of 2 PETSc finalized with errcode= 0 @ 0 goodbye from 0 of 2 -------------------------------------------------------------------------- The call to cuEventDestory failed. This is a unrecoverable error and will cause the program to abort. cuEventDestory return value: 400 Check the cuda.h file for what the return value means. -------------------------------------------------------------------------- -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec noticed that process rank 0 with PID 0 on node XPS-15 exited on signal 11 (Segmentation fault). -------------------------------------------------------------------------- [XPS-15:23232] 127 more processes have sent help message help-mpi-common-cuda.txt / cuEventCreate failed [XPS-15:23232] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [XPS-15:23232] 127 more processes have sent help message help-mpi-common-cuda.txt / cuIpcGetEventHandle failed [XPS-15:23232] 149 more processes have sent help message help-mpi-common-cuda.txt / cuIpcGetMemHandle failed