<div dir="ltr"><div>I beg your burden, how can I use those tools of PETSc in PDSLin program ... </div><div><br></div><div>valgrind is working in such "mpirun -n $n valgrind ./dtest_pdslin $inputFIle $matrixFile " </div><div><br></div><div><span style="font-size:12.8px">but these options are not working</span></div><div><span style="font-size:12.8px"><br></span><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Smith, Barry F.</b> <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>></span><br>Date: 30 April 2018 at 18:39<br>Subject: Re: [petsc-users] PDSLin hybrid solver performance with PETSc<br>To: Afrah Najib <<a href="mailto:afrah.nacib@gmail.com">afrah.nacib@gmail.com</a>><br>Cc: PETSc <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>, "Xiaoye S. Li" <<a href="mailto:xsli@lbl.gov">xsli@lbl.gov</a>>, Ichitaro Yamazaki <<a href="mailto:iyamazak@icl.utk.edu">iyamazak@icl.utk.edu</a>>, Ichitaro Yamazaki <<a href="mailto:ic.yamazaki@gmail.com">ic.yamazaki@gmail.com</a>><br><br><br><br>
   Looks like there is a divide by a variable that is zero in pdslin where the code doesn't first check if the value is not zero. <br>
<br>
   You can try using the option -start_in_debugger and in each debugger window use the debugger option to catch floating point exceptions and then type cont to continue the run. Yes this is annoying when you have so many debugger windows but is the only simple way to determine immediately the line of code that is producing the floating point exception.<br>
<br>
   Barry<br>
<div><div class="gmail-h5"><br>
<br>
> On Apr 30, 2018, at 6:20 AM, Afrah Najib <<a href="mailto:afrah.nacib@gmail.com">afrah.nacib@gmail.com</a>> wrote:<br>
> <br>
> Hi all,<br>
> <br>
> I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows)<br>
> <a href="https://sparse.tamu.edu/GHS_psdef/audikw_1" rel="noreferrer" target="_blank">https://sparse.tamu.edu/GHS_<wbr>psdef/audikw_1</a><br>
> <br>
> I am also running on cluster with  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node<br>
> <br>
> but it takes a lot of time for Schur complement factorization. <br>
> <br>
> Compiler      Subdomains      processors      Precond TmatrixPartition        TmatrixDist     Tfactorise      TcompApproSchur TprepApproSchur Tfactoschur     Tprecond_Total  Tsolve<br>
> gcc   8       128     Sparse  23.49532        0.6009915       6.059546        212.8243        0.04027608      38.74885        257.6796        13.69609<br>
> gcc   8       16      Sparse  11.864  0.4385703       27.23499        1390.153        0.5523289       55.20706        1473.207        4.213825<br>
> gcc   8       32      Sparse  9.912792        0.2592737       16.79125        934.3571        0.2372452       49.39175        1000.813        37.2029<br>
> gcc   8       64      Sparse  12.61929        0.2336042       8.2316  632.839 0.1113939       52.21551        693.4153        13.94034<br>
> <br>
> My PETSc in stallation configuration is like this:<br>
> <br>
> export PETSC_DIR=/X/petsc<br>
> export PETSC_ARCH=arch-real<br>
> <br>
> ./configure PETSC_DIR=/X/petsc \<br>
> prefix=/X/petsc/petsc-install \<br>
> PETSC_ARCH=arch-real  \<br>
> --with-debugging=0 \<br>
> --with-cc=/Y/progs/openmpi/<wbr>openmpi-3.0.0-gcc-7.1.0/bin/<wbr>mpicc \<br>
> --with-cxx=/Y/progs/openmpi/<wbr>openmpi-3.0.0-gcc-7.1.0/bin/<wbr>mpicxx \<br>
> --with-fc=/Y/progs/openmpi/<wbr>openmpi-3.0.0-gcc-7.1.0/bin/<wbr>mpif90 \<br>
> --with-valgrind=1 \<br>
> --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 \<br>
> --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 \<br>
> --with-blas-lapack-dir=/Y/<wbr>progs/intel/compilers_and_<wbr>libraries_2017.0.098/linux/<wbr>mkl/lib/intel64.<br>
> <br>
> My slurm scrpt content is like this:<br>
> <br>
> #!/bin/bash -l<br>
> #SBATCH -J "Paudik16"<br>
> #SBATCH -A X<br>
> #SBATCH -p X<br>
> #SBATCH --exclusive<br>
> #SBATCH -o Paudik_8_2_16.%j.out<br>
> #SBATCH -e Paudik_8_2_16.%j.err<br>
> <br>
> #SBATCH -N 8<br>
> #SBATCH --ntasks-per-node=2<br>
> #SBATCH --cpus-per-task=1<br>
> <br>
> export KMP_AFFINITY=verbose,disabled<br>
> export MKL_NUM_THREADS=${SLURM_CPUS_<wbr>PER_TASK}<br>
> export OMP_NUM_THREADS=${SLURM_CPUS_<wbr>PER_TASK}<br>
> <br>
> module purge<br>
> <br>
> module load gcc/7.1.0<br>
> module load mpi/openmpi-3.0.0-gcc-7.1.0<br>
> module load intel/mkl_2017.0.098<br>
> <br>
> file=<a href="http://input.in" rel="noreferrer" target="_blank">input.in</a><br>
> matrixFile=~/dataDir/ijv/<wbr>audikw_1.ijv<br>
> <br>
> mpirun -np  $SLURM_NTASKS ./dtest_pdslin $file $matrixFile<br>
> <br>
> <br>
> <br>
> I attached an output of one file along with the input configuration of PDSLin.<br>
> <br>
> Any idea of how to enhance these results.<br>
> <br>
> Note  in PDSLin, I tried with parallel Schur permutation but I got the following error :<br>
> <br>
> [1]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [1]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [1]PETSC ERROR: to get more information on the crash.<br>
> [1]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> ------------------------------<wbr>------------------------------<wbr>--------------<br>
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD<br>
> with errorcode 59.<br>
> <br>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.<br>
> You may or may not see output from other processes, depending on<br>
> e/Xactly when Open MPI kills them.<br>
> ------------------------------<wbr>------------------------------<wbr>--------------<br>
> [13]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [13]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [13]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [13]PETSC ERROR: to get more information on the crash.<br>
> [11]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [11]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [11]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [11]PETSC ERROR: to get more information on the crash.<br>
> [3]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [3]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [3]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [3]PETSC ERROR: to get more information on the crash.<br>
> [3]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [3]PETSC ERROR: Signal received<br>
> [3]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [9]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [9]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [9]PETSC ERROR: to get more information on the crash.<br>
> [7]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [7]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [7]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [7]PETSC ERROR: to get more information on the crash.<br>
> [5]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [5]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [5]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [5]PETSC ERROR: to get more information on the crash.<br>
> [5]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [5]PETSC ERROR: Signal received<br>
> [5]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero<br>
> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [15]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [13]PETSC ERROR: Signal received<br>
> [13]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [13]PETSC ERROR: ./dtest_pdslin on a  named s060 by afarea Mon Apr 30 05:29:53 2018<br>
> [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [11]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [11]PETSC ERROR: Signal received<br>
> [11]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [11]PETSC ERROR: ./dtest_pdslin on a  named s054 by afarea Mon Apr 30 05:29:53 2018<br>
> [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> ./dtest_pdslin on a  named s034 by afarea Mon Apr 30 05:29:53 2018<br>
> [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [3]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [9]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [9]PETSC ERROR: Signal received<br>
> [9]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [9]PETSC ERROR: ./dtest_pdslin on a  named s044 by afarea Mon Apr 30 05:29:53 2018<br>
> [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [7]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [7]PETSC ERROR: Signal received<br>
> [7]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [7]PETSC ERROR: ./dtest_pdslin on a  named s036 by afarea Mon Apr 30 05:29:53 2018<br>
> [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [7]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> ./dtest_pdslin on a  named s035 by afarea Mon Apr 30 05:29:53 2018<br>
> [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [5]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> configure using --with-debugging=yes, recompile, link, and run<br>
> [15]PETSC ERROR: to get more information on the crash.<br>
> [15]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [15]PETSC ERROR: Signal received<br>
> [15]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [15]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30 05:29:53 2018<br>
> [13]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [11]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [9]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [15]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [1]PETSC ERROR: Signal received<br>
> [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [1]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30 05:29:53 2018<br>
> [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [1]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30 05:29:53 2018<br>
> [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [6]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [6]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [6]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [6]PETSC ERROR: to get more information on the crash.<br>
> [6]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [6]PETSC ERROR: Signal received<br>
> [6]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [6]PETSC ERROR: ./dtest_pdslin on a  named s036 by afarea Mon Apr 30 05:29:53 2018<br>
> [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [6]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [4]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [4]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [4]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [4]PETSC ERROR: to get more information on the crash.<br>
> [4]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [4]PETSC ERROR: Signal received<br>
> [4]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [4]PETSC ERROR: ./dtest_pdslin on a  named s035 by afarea Mon Apr 30 05:29:53 2018<br>
> [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [4]PETSC ERROR: ./dtest_pdslin on a  named s035 by afarea Mon Apr 30 05:29:53 2018<br>
> [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [4]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [0]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [0]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [0]PETSC ERROR: to get more information on the crash.<br>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [0]PETSC ERROR: Signal received<br>
> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [0]PETSC ERROR: ./dtest_pdslin on a  named f002 by afarea Mon Apr 30 05:29:53 2018<br>
> [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [10]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [10]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [10]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [10]PETSC ERROR: to get more information on the crash.<br>
> [10]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [10]PETSC ERROR: Signal received<br>
> [10]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [10]PETSC ERROR: ./dtest_pdslin on a  named s054 by afarea Mon Apr 30 05:29:53 2018<br>
> <br>
> [8]PETSC ERROR: ./dtest_pdslin on a  named s044 by afarea Mon Apr 30 05:29:53 2018<br>
> [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [8]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [14]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [14]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [14]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [14]PETSC ERROR: to get more information on the crash.<br>
> [14]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [14]PETSC ERROR: Signal received<br>
> [14]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [14]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30 05:29:53 2018<br>
> [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [14]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [15]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [15]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [15]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [15]PETSC ERROR: to get more information on the crash.<br>
> [15]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [15]PETSC ERROR: Signal received<br>
> [15]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [15]PETSC ERROR: ./dtest_pdslin on a  named s071 by afarea Mon Apr 30 05:29:53 2018<br>
> [15]PETSC ERROR: #2 User provided function() line 0 in  unknown file<br>
> [2]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [2]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [2]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [2]PETSC ERROR: to get more information on the crash.<br>
> [2]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [2]PETSC ERROR: Signal received<br>
> [2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [2]PETSC ERROR: ./dtest_pdslin on a  named s034 by afarea Mon Apr 30 05:29:53 2018<br>
> [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [2]PETSC ERROR: #1 User provided function() line 0 in  unknown file<br>
> [3]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
> [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [3]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html#<wbr>valgrind</a><br>
> [3]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors<br>
> [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>
> [3]PETSC ERROR: to get more information on the crash.<br>
> [3]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> [3]PETSC ERROR: Signal received<br>
> [3]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
> [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown<br>
> [3]PETSC ERROR: ./dtest_pdslin on a  named s034 by afarea Mon Apr 30 05:29:53 2018<br>
> [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpicc --with-c/X/X=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpic/X/X --with-fc=/okyanus/progs/<wbr>openmpi/openmpi-3.0.0-gcc-7.1.<wbr>0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/<wbr>usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/<wbr>local/hwloc-1.11.9 --with-blaslapack-dir=/<wbr>okyanus/progs/intel/compilers_<wbr>and_libraries_2017.0.098/linu/<wbr>X/mkl/lib/intel64<br>
> [3]PETSC ERROR: #2 User provided function() line 0 in  unknown file<br>
> [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort<br>
> [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages<br>
> <br>
> <br>
</div></div>> <Paudik_8_2_16.100234.out><br>
<br>
</div><br></div><div><br></div></div>