<div dir="ltr">Hi all,<div><br></div><div>I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows)</div><div><a href="https://sparse.tamu.edu/GHS_psdef/audikw_1">https://sparse.tamu.edu/GHS_psdef/audikw_1</a><br></div><div><br></div><div>I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node</div><div><br></div><div>but it takes a lot of time for Schur complement factorization. </div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div><table cellspacing="0" border="0" style="font-family:"Liberation Sans";font-size:x-small">
<colgroup width="64"></colgroup>
<colgroup width="86"></colgroup>
<colgroup width="74"></colgroup>
<colgroup width="61"></colgroup>
<colgroup width="106"></colgroup>
<colgroup width="81"></colgroup>
<colgroup width="71"></colgroup>
<colgroup width="121"></colgroup>
<colgroup width="116"></colgroup>
<colgroup width="83"></colgroup>
<colgroup width="67"></colgroup>
<colgroup width="66"></colgroup>
<tbody style="font-family:"Liberation Sans";font-size:x-small"><tr style="font-family:"Liberation Sans";font-size:x-small">
<td height="17" align="center" style="font-family:"Liberation Sans";font-size:x-small">Compiler</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">Subdomains</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">processors</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">Precond</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">TmatrixPartition</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">TmatrixDist</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">Tfactorise</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">TcompApproSchur</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">TprepApproSchur</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">Tfactoschur</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">Tprecond_Total</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small"> Tsolve</td>
</tr>
<tr style="font-family:"Liberation Sans";font-size:x-small">
<td height="17" align="center" style="font-family:"Liberation Sans";font-size:x-small">gcc</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">8</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">128</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small"> Sparse</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">23.49532</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.6009915</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">6.059546</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">212.8243</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.04027608</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">38.74885</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">257.6796</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">13.69609</td>
</tr>
<tr style="font-family:"Liberation Sans";font-size:x-small">
<td height="17" align="center" style="font-family:"Liberation Sans";font-size:x-small">gcc</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">8</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">16</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small"> Sparse</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">11.864</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.4385703</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">27.23499</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">1390.153</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.5523289</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">55.20706</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">1473.207</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">4.213825</td>
</tr>
<tr style="font-family:"Liberation Sans";font-size:x-small">
<td height="17" align="center" style="font-family:"Liberation Sans";font-size:x-small">gcc</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">8</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">32</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small"> Sparse</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">9.912792</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.2592737</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">16.79125</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">934.3571</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.2372452</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">49.39175</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">1000.813</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">37.2029</td>
</tr>
<tr style="font-family:"Liberation Sans";font-size:x-small">
<td height="17" align="center" style="font-family:"Liberation Sans";font-size:x-small">gcc</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">8</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">64</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small"> Sparse</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">12.61929</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.2336042</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">8.2316</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">632.839</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">0.1113939</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">52.21551</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">693.4153</td>
<td align="center" style="font-family:"Liberation Sans";font-size:x-small">13.94034</td>
</tr>
</tbody></table></div></blockquote></div><div>
<span></span>
<br></div><div>My PETSc in stallation configuration is like this:</div><div><br></div><div><div>export PETSC_DIR=/X/petsc</div><div>export PETSC_ARCH=arch-real</div><div><br></div><div>./configure PETSC_DIR=/X/petsc \</div><div>prefix=/X/petsc/petsc-install \</div><div>PETSC_ARCH=arch-real \</div><div>--with-debugging=0 \</div><div>--with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \</div><div>--with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \</div><div>--with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \</div><div>--with-valgrind=1 \</div><div>--with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \</div><div>--with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \</div><div>--with-blas-lapack-dir=/Y/progs/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64.</div></div><div><br></div><div><b>My slurm scrpt content is like this:</b></div><div><b><br></b></div><div><div>#!/bin/bash -l</div><div>#SBATCH -J "Paudik16"</div><div>#SBATCH -A X</div><div>#SBATCH -p X</div><div>#SBATCH --exclusive</div><div>#SBATCH -o Paudik_8_2_16.%j.out</div><div>#SBATCH -e Paudik_8_2_16.%j.err</div><div><br></div><div>#SBATCH -N 8</div><div>#SBATCH --ntasks-per-node=2</div><div>#SBATCH --cpus-per-task=1</div><div><br></div><div>export KMP_AFFINITY=verbose,disabled</div><div>export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}</div><div>export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}</div><div><br></div><div>module purge</div><div><br></div><div>module load gcc/7.1.0</div><div>module load mpi/openmpi-3.0.0-gcc-7.1.0</div><div>module load intel/mkl_2017.0.098</div><div><br></div><div>file=<a href="http://input.in">input.in</a></div><div>matrixFile=~/dataDir/ijv/audikw_1.ijv</div><div><br></div><div>mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile</div></div><div><br></div><div><br></div><div><br></div><div><b>I attached an output of one file along with the input configuration of PDSLin.</b></div><div><br></div><div>Any idea of how to enhance these results.</div><div><br></div><div>Note in PDSLin, I tried with parallel Schur permutation but I got the following error :</div><div><br></div><div><div>[1]PETSC ERROR: ------------------------------------------------------------------------</div><div>[1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[1]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[1]PETSC ERROR: to get more information on the crash.</div><div>[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>--------------------------------------------------------------------------</div><div>MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD</div><div>with errorcode 59.</div><div><br></div><div>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</div><div>You may or may not see output from other processes, depending on</div><div>e/Xactly when Open MPI kills them.</div><div>--------------------------------------------------------------------------</div><div>[13]PETSC ERROR: ------------------------------------------------------------------------</div><div>[13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[13]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[13]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[13]PETSC ERROR: to get more information on the crash.</div><div>[11]PETSC ERROR: ------------------------------------------------------------------------</div><div>[11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[11]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[11]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[11]PETSC ERROR: to get more information on the crash.</div><div>[3]PETSC ERROR: ------------------------------------------------------------------------</div><div>[3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[3]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[3]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[3]PETSC ERROR: to get more information on the crash.</div><div>[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[3]PETSC ERROR: Signal received</div><div>[3]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[3]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[3]PETSC ERROR: [9]PETSC ERROR: ------------------------------------------------------------------------</div><div>[9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[9]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[9]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[9]PETSC ERROR: to get more information on the crash.</div><div>[7]PETSC ERROR: ------------------------------------------------------------------------</div><div>[7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[7]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[7]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[7]PETSC ERROR: to get more information on the crash.</div><div>[5]PETSC ERROR: ------------------------------------------------------------------------</div><div>[5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[5]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[5]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[5]PETSC ERROR: to get more information on the crash.</div><div>[5]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[5]PETSC ERROR: Signal received</div><div>[5]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[5]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[5]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------</div><div>[15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero</div><div>[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[15]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[13]PETSC ERROR: Signal received</div><div>[13]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[13]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 05:29:53 2018</div><div>[13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[11]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[11]PETSC ERROR: Signal received</div><div>[11]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[11]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018</div><div>[11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018</div><div>[3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[3]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[9]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[9]PETSC ERROR: Signal received</div><div>[9]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[9]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018</div><div>[9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[7]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[7]PETSC ERROR: Signal received</div><div>[7]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[7]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018</div><div>[7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[7]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018</div><div>[5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[5]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>configure using --with-debugging=yes, recompile, link, and run</div><div>[15]PETSC ERROR: to get more information on the crash.</div><div>[15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[15]PETSC ERROR: Signal received</div><div>[15]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[15]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018</div><div>[13]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[11]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[9]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[15]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[1]PETSC ERROR: Signal received</div><div>[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[1]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018</div><div>[1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[1]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[1]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018</div><div>[1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[1]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[6]PETSC ERROR: ------------------------------------------------------------------------</div><div>[6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[6]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[6]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[6]PETSC ERROR: to get more information on the crash.</div><div>[6]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[6]PETSC ERROR: Signal received</div><div>[6]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[6]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018</div><div>[6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[6]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[4]PETSC ERROR: ------------------------------------------------------------------------</div><div>[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[4]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[4]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[4]PETSC ERROR: to get more information on the crash.</div><div>[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[4]PETSC ERROR: Signal received</div><div>[4]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[4]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018</div><div>[4]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018</div><div>[4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[4]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[0]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[0]PETSC ERROR: to get more information on the crash.</div><div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[0]PETSC ERROR: Signal received</div><div>[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[0]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018</div><div>[0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[0]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[10]PETSC ERROR: ------------------------------------------------------------------------</div><div>[10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[10]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[10]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[10]PETSC ERROR: to get more information on the crash.</div><div>[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[10]PETSC ERROR: Signal received</div><div>[10]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[10]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018</div><div><br></div><div>[8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018</div><div>[8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[8]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[14]PETSC ERROR: ------------------------------------------------------------------------</div><div>[14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[14]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[14]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[14]PETSC ERROR: to get more information on the crash.</div><div>[14]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[14]PETSC ERROR: Signal received</div><div>[14]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[14]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018</div><div>[14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[14]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[15]PETSC ERROR: ------------------------------------------------------------------------</div><div>[15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[15]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[15]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[15]PETSC ERROR: to get more information on the crash.</div><div>[15]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[15]PETSC ERROR: Signal received</div><div>[15]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[15]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018</div><div>[15]PETSC ERROR: #2 User provided function() line 0 in unknown file</div><div>[2]PETSC ERROR: ------------------------------------------------------------------------</div><div>[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[2]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[2]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[2]PETSC ERROR: to get more information on the crash.</div><div>[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[2]PETSC ERROR: Signal received</div><div>[2]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[2]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018</div><div>[2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[2]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div>[3]PETSC ERROR: ------------------------------------------------------------------------</div><div>[3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div>[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div>[3]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div>[3]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linu/X and Apple Mac OS /X to find memory corruption errors</div><div>[3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run</div><div>[3]PETSC ERROR: to get more information on the crash.</div><div>[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div>[3]PETSC ERROR: Signal received</div><div>[3]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</div><div>[3]PETSC ERROR: Petsc Release Version 3.9.0, unknown</div><div>[3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018</div><div>[3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64</div><div>[3]PETSC ERROR: #2 User provided function() line 0 in unknown file</div><div>[f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort</div><div>[f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</div><div><br></div></div><div><br></div></div>