[petsc-users] PDSLin hybrid solver performance with PETSc

Mon Apr 30 06:41:11 CDT 2018

On Mon, Apr 30, 2018 at 7:20 AM, Afrah Najib <afrah.nacib at gmail.com> wrote:

> Hi all,
>
> I am testing PDSLin hybrid solver performance with PETSc krylov method on
> a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by
> rows)
> https://sparse.tamu.edu/GHS_psdef/audikw_1
>
> I am also running on cluster with  Intel(R) Xeon(R) CPU E5-2680 v4 @
> 2.40GHz processor and 28 core , 128 ; 512GB per node
>

1) For any solver performance question, we need to see the output of

  -log_view -ksp_monitor_true_residual -ksp_converged_reason -ksp_view

2) Without knowing how many nodes (not cores) you are running on, we cannot
evaluate the speedup, see

  http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

3) Your SEGV is not occurring in a PETSc function. If it were, the signal
handler would report the function name.

  Thanks,

    Matt

but it takes a lot of time for Schur complement factorization.
>
>
> Compiler Subdomains processors Precond TmatrixPartition TmatrixDist
> Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total
> Tsolve
> gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885
> 257.6796 13.69609
> gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706
> 1473.207 4.213825
> gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175
> 1000.813 37.2029
> gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551
> 693.4153 13.94034
>
>
> My PETSc in stallation configuration is like this:
>
> export PETSC_DIR=/X/petsc
> export PETSC_ARCH=arch-real
>
> ./configure PETSC_DIR=/X/petsc \
> prefix=/X/petsc/petsc-install \
> PETSC_ARCH=arch-real  \
> --with-debugging=0 \
> --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \
> --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \
> --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \
> --with-valgrind=1 \
> --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \
> --with-blas-lapack-dir=/Y/progs/intel/compilers_and_
> libraries_2017.0.098/linux/mkl/lib/intel64.
>
> *My slurm scrpt content is like this:*
>
> #!/bin/bash -l
> #SBATCH -J "Paudik16"
> #SBATCH -A X
> #SBATCH -p X
> #SBATCH --exclusive
> #SBATCH -o Paudik_8_2_16.%j.out
> #SBATCH -e Paudik_8_2_16.%j.err
>
> #SBATCH -N 8
> #SBATCH --ntasks-per-node=2
> #SBATCH --cpus-per-task=1
>
> export KMP_AFFINITY=verbose,disabled
> export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}
> export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
>
> module purge
>
> module load gcc/7.1.0
> module load mpi/openmpi-3.0.0-gcc-7.1.0
> module load intel/mkl_2017.0.098
>
> file=input.in
> matrixFile=~/dataDir/ijv/audikw_1.ijv
>
> mpirun -np  $SLURM_NTASKS ./dtest_pdslin $file $matrixFile
>
>
>
> *I attached an output of one file along with the input configuration of
> PDSLin.*
>
> Any idea of how to enhance these results.
>
> Note  in PDSLin, I tried with parallel Schur permutation but I got the
> following error :
>
> [1]PETSC ERROR: ------------------------------
> ------------------------------------------
> [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [1]PETSC ERROR: to get more information on the crash.
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> e/Xactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [13]PETSC ERROR: ------------------------------
> ------------------------------------------
> [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac
> OS /X to find memory corruption errors
> [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [13]PETSC ERROR: to get more information on the crash.
> [11]PETSC ERROR: ------------------------------
> ------------------------------------------
> [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac
> OS /X to find memory corruption errors
> [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [11]PETSC ERROR: to get more information on the crash.
> [3]PETSC ERROR: ------------------------------
> ------------------------------------------
> [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [3]PETSC ERROR: to get more information on the crash.
> [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3]PETSC ERROR: Signal received
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------
> ------------------------------------------
> [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [9]PETSC ERROR: to get more information on the crash.
> [7]PETSC ERROR: ------------------------------
> ------------------------------------------
> [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [7]PETSC ERROR: to get more information on the crash.
> [5]PETSC ERROR: ------------------------------
> ------------------------------------------
> [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [5]PETSC ERROR: to get more information on the crash.
> [5]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [5]PETSC ERROR: Signal received
> [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------
> ------------------------------------------
> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point
> E/Xception,probably divide by zero
> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [13]PETSC ERROR: Signal received
> [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [13]PETSC ERROR: ./dtest_pdslin on a  named s060 by afarea Mon Apr 30
> 05:29:53 2018
> [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [11]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [11]PETSC ERROR: Signal received
> [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [11]PETSC ERROR: ./dtest_pdslin on a  named s054 by afarea Mon Apr 30
> 05:29:53 2018
> [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> ./dtest_pdslin on a  named s034 by afarea Mon Apr 30 05:29:53 2018
> [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [3]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [9]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [9]PETSC ERROR: Signal received
> [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [9]PETSC ERROR: ./dtest_pdslin on a  named s044 by afarea Mon Apr 30
> 05:29:53 2018
> [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [7]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [7]PETSC ERROR: Signal received
> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [7]PETSC ERROR: ./dtest_pdslin on a  named s036 by afarea Mon Apr 30
> 05:29:53 2018
> [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [7]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> ./dtest_pdslin on a  named s035 by afarea Mon Apr 30 05:29:53 2018
> [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [5]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> configure using --with-debugging=yes, recompile, link, and run
> [15]PETSC ERROR: to get more information on the crash.
> [15]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [15]PETSC ERROR: Signal received
> [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [15]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30
> 05:29:53 2018
> [13]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [11]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [9]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [15]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [1]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30
> 05:29:53 2018
> [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [1]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30
> 05:29:53 2018
> [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [6]PETSC ERROR: ------------------------------
> ------------------------------------------
> [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [6]PETSC ERROR: to get more information on the crash.
> [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [6]PETSC ERROR: Signal received
> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [6]PETSC ERROR: ./dtest_pdslin on a  named s036 by afarea Mon Apr 30
> 05:29:53 2018
> [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [6]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [4]PETSC ERROR: ------------------------------
> ------------------------------------------
> [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [4]PETSC ERROR: to get more information on the crash.
> [4]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [4]PETSC ERROR: Signal received
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [4]PETSC ERROR: ./dtest_pdslin on a  named s035 by afarea Mon Apr 30
> 05:29:53 2018
> [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [4]PETSC ERROR: ./dtest_pdslin on a  named s035 by afarea Mon Apr 30
> 05:29:53 2018
> [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [0]PETSC ERROR: ------------------------------
> ------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [0]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30
> 05:29:53 2018
> [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [10]PETSC ERROR: ------------------------------
> ------------------------------------------
> [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac
> OS /X to find memory corruption errors
> [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [10]PETSC ERROR: to get more information on the crash.
> [10]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [10]PETSC ERROR: Signal received
> [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [10]PETSC ERROR: ./dtest_pdslin on a  named s054 by afarea Mon Apr 30
> 05:29:53 2018
>
> [8]PETSC ERROR: ./dtest_pdslin on a  named s044 by afarea Mon Apr 30
> 05:29:53 2018
> [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [8]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [14]PETSC ERROR: ------------------------------
> ------------------------------------------
> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac
> OS /X to find memory corruption errors
> [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [14]PETSC ERROR: to get more information on the crash.
> [14]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [14]PETSC ERROR: Signal received
> [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [14]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30
> 05:29:53 2018
> [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [14]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [15]PETSC ERROR: ------------------------------
> ------------------------------------------
> [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac
> OS /X to find memory corruption errors
> [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> and run
> [15]PETSC ERROR: to get more information on the crash.
> [15]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [15]PETSC ERROR: Signal received
> [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [15]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30
> 05:29:53 2018
> [15]PETSC ERROR: #2 User provided function() line 0 in  unknown file
> [2]PETSC ERROR: ------------------------------
> ------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [2]PETSC ERROR: to get more information on the crash.
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: Signal received
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [2]PETSC ERROR: ./dtest_pdslin on a  named s034 by afarea Mon Apr 30
> 05:29:53 2018
> [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [2]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> [3]PETSC ERROR: ------------------------------
> ------------------------------------------
> [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/
> documentation/faq.html#valgrind
> [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS
> /X to find memory corruption errors
> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [3]PETSC ERROR: to get more information on the crash.
> [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3]PETSC ERROR: Signal received
> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown
> [3]PETSC ERROR: ./dtest_pdslin on a  named s034 by afarea Mon Apr 30
> 05:29:53 2018
> [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc
> prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0
> --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc
> --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X
> --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90
> --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0
> --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/
> okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/
> X/mkl/lib/intel64
> [3]PETSC ERROR: #2 User provided function() line 0 in  unknown file
> [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt /
> mpi-abort
> [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
> help / error messages
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180430/fbb07e99/attachment-0001.html>