<div dir="ltr">Ok, last output was from simulated multicores, in an actual cluster the errors are of the kind:<div><br></div><div><div><font face="monospace, monospace">[valera@cinci CSRMatrix]$ petsc -n 2 ./solvelinearmgPETSc</font></div><div><font face="monospace, monospace"> TrivSoln loaded, size: 4 / 4</font></div><div><font face="monospace, monospace"> TrivSoln loaded, size: 4 / 4</font></div><div><font face="monospace, monospace"> RHS loaded, size: 4 / 4</font></div><div><font face="monospace, monospace"> RHS loaded, size: 4 / 4</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Argument out of range</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Comm must be of size 1</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[0]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Argument out of range</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Comm must be of size 1</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #1 MatCreate_SeqAIJ() line 3958 in /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #2 MatSetType() line 94 in /home/valera/petsc-3.7.2/src/mat/interface/matreg.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #3 MatCreateSeqAIJWithArrays() line 4300 in /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace"> local size: 2</font></div><div><font face="monospace, monospace"> local size: 2</font></div><div><font face="monospace, monospace">Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #1 MatCreate_SeqAIJ() line 3958 in /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #2 MatSetType() line 94 in /home/valera/petsc-3.7.2/src/mat/interface/matreg.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #3 MatCreateSeqAIJWithArrays() line 4300 in /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: Nonconforming object sizes</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Sum of local lengths 8 does not equal global length 4, my local length 4</font></div><div><font face="monospace, monospace"> likely a call to VecSetSizes() or MatSetSizes() is wrong.</font></div><div><font face="monospace, monospace">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#split">http://www.mcs.anl.gov/petsc/documentation/faq.html#split</a></font></div><div><font face="monospace, monospace">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">Nonconforming object sizes</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Sum of local lengths 8 does not equal global length 4, my local length 4</font></div><div><font face="monospace, monospace"> likely a call to VecSetSizes() or MatSetSizes() is wrong.</font></div><div><font face="monospace, monospace">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#split">http://www.mcs.anl.gov/petsc/documentation/faq.html#split</a></font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[0]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #4 PetscSplitOwnership() line 93 in /home/valera/petsc-3.7.2/src/sys/utils/psplit.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #4 PetscSplitOwnership() line 93 in /home/valera/petsc-3.7.2/src/sys/utils/psplit.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #5 PetscLayoutSetUp() line 143 in /home/valera/petsc-3.7.2/src/vec/is/utils/pmap.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #6 MatMPIAIJSetPreallocation_MPIAIJ() line 2768 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #5 PetscLayoutSetUp() line 143 in /home/valera/petsc-3.7.2/src/vec/is/utils/pmap.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: #7 MatMPIAIJSetPreallocation() line 3505 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">#6 MatMPIAIJSetPreallocation_MPIAIJ() line 2768 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: #8 MatSetUp_MPIAIJ() line 2153 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">#7 MatMPIAIJSetPreallocation() line 3505 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #8 MatSetUp_MPIAIJ() line 2153 in /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #9 MatSetUp() line 739 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #9 MatSetUp() line 739 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Object is in wrong state</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatSetNearNullSpace()</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[0]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">Object is in wrong state</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatSetNearNullSpace()</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: #10 MatSetNearNullSpace() line 8195 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #10 MatSetNearNullSpace() line 8195 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Object is in wrong state</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatAssemblyBegin()</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: Object is in wrong state</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Must call MatXXXSetPreallocation() or MatSetUp() on argument 1 "mat" before MatAssemblyBegin()</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #11 MatAssemblyBegin() line 5093 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #11 MatAssemblyBegin() line 5093 in /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: ------------------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ------------------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></font></div><div><font face="monospace, monospace">[1]PETSC ERROR: or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</font></div><div><font face="monospace, monospace">or try <a href="http://valgrind.org">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: likely location of problem given in stack below</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: likely location of problem given in stack below</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: INSTEAD the line number of the start of the function</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: INSTEAD the line number of the start of the function</font></div><div><font face="monospace, monospace"> is given.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatAssemblyEnd line 5185 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: is given.</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatAssemblyEnd line 5185 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0] MatAssemblyBegin line 5090 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatSetNearNullSpace line 8191 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: [1] MatAssemblyBegin line 5090 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0] PetscSplitOwnership line 80 /home/valera/petsc-3.7.2/src/sys/utils/psplit.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] PetscLayoutSetUp line 129 /home/valera/petsc-3.7.2/src/vec/is/utils/pmap.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatMPIAIJSetPreallocation_MPIAIJ line 2767 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1] MatSetNearNullSpace line 8191 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] PetscSplitOwnership line 80 /home/valera/petsc-3.7.2/src/sys/utils/psplit.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [0]PETSC ERROR: [0] MatMPIAIJSetPreallocation line 3502 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatSetUp_MPIAIJ line 2152 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1] PetscLayoutSetUp line 129 /home/valera/petsc-3.7.2/src/vec/is/utils/pmap.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatMPIAIJSetPreallocation_MPIAIJ line 2767 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatSetUp line 727 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatCreate_SeqAIJ line 3956 /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatMPIAIJSetPreallocation line 3502 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatSetUp_MPIAIJ line 2152 /home/valera/petsc-3.7.2/src/mat/impls/aij/mpi/mpiaij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatSetType line 44 /home/valera/petsc-3.7.2/src/mat/interface/matreg.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [0] MatCreateSeqAIJWithArrays line 4295 /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatSetUp line 727 /home/valera/petsc-3.7.2/src/mat/interface/matrix.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatCreate_SeqAIJ line 3956 /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Signal received</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatSetType line 44 /home/valera/petsc-3.7.2/src/mat/interface/matreg.c</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: [1] MatCreateSeqAIJWithArrays line 4295 /home/valera/petsc-3.7.2/src/mat/impls/aij/seq/aij.c</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[0]PETSC ERROR: Signal received</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #12 User provided function() line 0 in unknown file</font></div><div><font face="monospace, monospace">Petsc Release Version 3.7.2, Jun, 05, 2016 </font></div><div><font face="monospace, monospace">[1]PETSC ERROR: ./solvelinearmgPETSc P on a arch-linux2-c-debug named cinci by valera Mon Sep 26 16:39:02 2016</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich</font></div><div><font face="monospace, monospace">[1]PETSC ERROR: #12 User provided function() line 0 in unknown file</font></div><div><font face="monospace, monospace">application called MPI_Abort(comm=0x84000004, 59) - process 0</font></div><div><font face="monospace, monospace">[cli_0]: aborting job:</font></div><div><font face="monospace, monospace">application called MPI_Abort(comm=0x84000004, 59) - process 0</font></div><div><font face="monospace, monospace">application called MPI_Abort(comm=0x84000002, 59) - process 1</font></div><div><font face="monospace, monospace">[cli_1]: aborting job:</font></div><div><font face="monospace, monospace">application called MPI_Abort(comm=0x84000002, 59) - process 1</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">===================================================================================</font></div><div><font face="monospace, monospace">= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</font></div><div><font face="monospace, monospace">= PID 10266 RUNNING AT cinci</font></div><div><font face="monospace, monospace">= EXIT CODE: 59</font></div><div><font face="monospace, monospace">= CLEANING UP REMAINING PROCESSES</font></div><div><font face="monospace, monospace">= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</font></div><div><font face="monospace, monospace">===================================================================================</font></div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 26, 2016 at 3:51 PM, Manuel Valera <span dir="ltr"><<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ok, i created a tiny testcase just for this,<div><br></div><div>The output from n# calls are as follows:</div><div><br></div><div><font face="monospace, monospace">n1:</font></div><div><span class=""><div><font face="monospace, monospace">Mat Object: 1 MPI processes</font></div><div><font face="monospace, monospace"> type: mpiaij</font></div></span><div><font face="monospace, monospace">row 0: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 1: (0, 2.) (1, 1.) (2, 3.) (3, 4.) </font></div><div><font face="monospace, monospace">row 2: (0, 4.) (1, 3.) (2, 1.) (3, 2.) </font></div><div><font face="monospace, monospace">row 3: (0, 3.) (1, 4.) (2, 2.) (3, 1.) </font></div></div><div><font face="monospace, monospace"><br></font></div><div><span style="font-family:monospace,monospace">n2:</span><br></div><div><div><font face="monospace, monospace">Mat Object: 2 MPI processes</font></div><div><font face="monospace, monospace"> type: mpiaij</font></div><div><font face="monospace, monospace">row 0: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 1: (0, 2.) (1, 1.) (2, 3.) (3, 4.) </font></div><div><font face="monospace, monospace">row 2: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 3: (0, 2.) (1, 1.) (2, 3.) (3, 4.) </font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">n4:</font></div><div><div><font face="monospace, monospace">Mat Object: 4 MPI processes</font></div><div><font face="monospace, monospace"> type: mpiaij</font></div><div><font face="monospace, monospace">row 0: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 1: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 2: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div><div><font face="monospace, monospace">row 3: (0, 1.) (1, 2.) (2, 4.) (3, 3.) </font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">It really gets messed, no idea what's happening.</font></div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 26, 2016 at 3:12 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
> On Sep 26, 2016, at 5:07 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
><br>
> Ok i was using a big matrix before, from a smaller testcase i got the output and effectively, it looks like is not well read at all, results are attached for DRAW viewer, output is too big to use STDOUT even in the small testcase. n# is the number of processors requested.<br>
<br>
</span> You need to construct a very small test case so you can determine why the values do not end up where you expect them. There is no way around it.<br>
<span>><br>
> is there a way to create the matrix in one node and the distribute it as needed on the rest ? maybe that would work.<br>
<br>
</span> No the is not scalable. You become limited by the memory of the one node.<br>
<div><div><br>
><br>
> Thanks<br>
><br>
> On Mon, Sep 26, 2016 at 2:40 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> How large is the matrix? It will take a very long time if the matrix is large. Debug with a very small matrix.<br>
><br>
> Barry<br>
><br>
> > On Sep 26, 2016, at 4:34 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> ><br>
> > Indeed there is something wrong with that call, it hangs out indefinitely showing only:<br>
> ><br>
> > Mat Object: 1 MPI processes<br>
> > type: mpiaij<br>
> ><br>
> > It draws my attention that this program works for 1 processor but not more, but it doesnt show anything for that viewer in either case.<br>
> ><br>
> > Thanks for the insight on the redundant calls, this is not very clear on documentation, which calls are included in others.<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Sep 26, 2016 at 2:02 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > The call to MatCreateMPIAIJWithArrays() is likely interpreting the values you pass in different than you expect.<br>
> ><br>
> > Put a call to MatView(Ap,PETSC_VIEWER_STDOUT<wbr>_WORLD,ierr) after the MatCreateMPIAIJWithArray() to see what PETSc thinks the matrix is.<br>
> ><br>
> ><br>
> > > On Sep 26, 2016, at 3:42 PM, Manuel Valera <<a href="mailto:mvalera@mail.sdsu.edu" target="_blank">mvalera@mail.sdsu.edu</a>> wrote:<br>
> > ><br>
> > > Hello,<br>
> > ><br>
> > > I'm working on solve a linear system in parallel, following ex12 of the ksp tutorial i don't see major complication on doing so, so for a working linear system solver with PCJACOBI and KSPGCR i did only the following changes:<br>
> > ><br>
> > > call MatCreate(PETSC_COMM_WORLD,Ap,<wbr>ierr)<br>
> > > ! call MatSetType(Ap,MATSEQAIJ,ierr)<br>
> > > call MatSetType(Ap,MATMPIAIJ,ierr) !paralellization<br>
> > ><br>
> > > call MatSetSizes(Ap,PETSC_DECIDE,PE<wbr>TSC_DECIDE,nbdp,nbdp,ierr);<br>
> > ><br>
> > > ! call MatSeqAIJSetPreallocationCSR(A<wbr>p,iapi,japi,app,ierr)<br>
> > > call MatSetFromOptions(Ap,ierr)<br>
> ><br>
> > Note that none of the lines above are needed (or do anything) because the MatCreateMPIAIJWithArrays() creates the matrix from scratch itself.<br>
> ><br>
> > Barry<br>
> ><br>
> > > ! call MatCreateSeqAIJWithArrays(PETS<wbr>C_COMM_WORLD,nbdp,nbdp,iapi,<wbr>japi,app,Ap,ierr)<br>
> > > call MatCreateMPIAIJWithArrays(PETS<wbr>C_COMM_WORLD,floor(real(nbdp)/<wbr>sizel),PETSC_DECIDE,nbdp,nbdp,<wbr>iapi,japi,app,Ap,ierr)<br>
> > ><br>
> > ><br>
> > > I grayed out the changes from sequential implementation.<br>
> > ><br>
> > > So, it does not complain at runtime until it reaches KSPSolve(), with the following error:<br>
> > ><br>
> > ><br>
> > > [1]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>--<br>
> > > [1]PETSC ERROR: Object is in wrong state<br>
> > > [1]PETSC ERROR: Matrix is missing diagonal entry 0<br>
> > > [1]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html</a> for trouble shooting.<br>
> > > [1]PETSC ERROR: Petsc Release Version 3.7.3, unknown<br>
> > > [1]PETSC ERROR: ./solvelinearmgPETSc � � on a arch-linux2-c-debug named valera-HP-xw4600-Workstation by valera Mon Sep 26 13:35:15 2016<br>
> > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack=1 --download-mpich=1 --download-ml=1<br>
> > > [1]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1733 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/mat/impls/aij/seq/aijfa<wbr>ct.c<br>
> > > [1]PETSC ERROR: #2 MatILUFactorSymbolic() line 6579 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/mat/interface/matrix.c<br>
> > > [1]PETSC ERROR: #3 PCSetUp_ILU() line 212 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/pc/impls/factor/ilu<wbr>/ilu.c<br>
> > > [1]PETSC ERROR: #4 PCSetUp() line 968 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/pc/interface/precon<wbr>.c<br>
> > > [1]PETSC ERROR: #5 KSPSetUp() line 390 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/ksp/interface/itfun<wbr>c.c<br>
> > > [1]PETSC ERROR: #6 PCSetUpOnBlocks_BJacobi_Single<wbr>block() line 650 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/pc/impls/bjacobi/<wbr>bjacobi.c<br>
> > > [1]PETSC ERROR: #7 PCSetUpOnBlocks() line 1001 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/pc/interface/precon<wbr>.c<br>
> > > [1]PETSC ERROR: #8 KSPSetUpOnBlocks() line 220 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/ksp/interface/itfun<wbr>c.c<br>
> > > [1]PETSC ERROR: #9 KSPSolve() line 600 in /home/valera/v5PETSc/petsc/pet<wbr>sc/src/ksp/ksp/interface/itfun<wbr>c.c<br>
> > > At line 333 of file solvelinearmgPETSc.f90<br>
> > > Fortran runtime error: Array bound mismatch for dimension 1 of array 'sol' (213120/106560)<br>
> > ><br>
> > ><br>
> > > This code works for -n 1 cores, but it gives this error when using more than one core.<br>
> > ><br>
> > > What am i missing?<br>
> > ><br>
> > > Regards,<br>
> > ><br>
> > > Manuel.<br>
> > ><br>
> > > <solvelinearmgPETSc.f90><br>
> ><br>
> ><br>
><br>
><br>
</div></div>> <n4.png><n2.png><n1.png><br>
<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>