On Fri, Aug 5, 2011 at 10:41 PM, Dominik Szczerba <span dir="ltr"><<a href="mailto:dominik@itis.ethz.ch">dominik@itis.ethz.ch</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I have a 2x6core. My solver works fine only on up to 8 processes,<br>
above that it always crashes with the below cited error. I did not yet<br>
valgrind etc. because I am in a desperate need to fix it quickly. I am<br>
just wondering what can potentially be the culprit.<br></blockquote><div><br></div><div>You are getting a SIGQUIT in a function you wrote (if it was a PETSc function it</div><div>would show up in the stack). It looks like the system might be killing your job.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
PS. I am not using MPI_Allreduce anywhere in my code.<br>
<br>
Many thanks for any hints,<br>
Dominik<br>
<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_9]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_1]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_7]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
INTERNAL ERROR: Invalid error class (66) encountered while returning from<br>
MPI_Allreduce. Please file a bug report. No error stack is available.<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_11]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
[0]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number 3 Quit: Some other process (or<br>
the batch system) has told this process to end<br>
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[0]PETSC ERROR: or see<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC</a><br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to<br>
find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>
[0]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[0]PETSC ERROR: is given.<br>
[0]PETSC ERROR: [0] MatAssemblyBegin_MPIAIJ line 462<br>
src/mat/impls/aij/mpi/mpiaij.c<br>
[0]PETSC ERROR: [0] MatAssemblyBegin line 4553 src/mat/interface/matrix.c<br>
[0]PETSC ERROR: [0] User provided functi[2]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[2]PETSC ERROR: Caught signal number 3 Quit: Some other process (or<br>
the batch system) has told this process to end<br>
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[2]PETSC ERROR: or see<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[2]PETSC</a><br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to<br>
find memory corruption errors<br>
[2]PETSC ERROR: likely location of problem given in stack below<br>
[2]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[2]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[2]PETSC ERROR: is given.<br>
[2]PETSC ERROR: [2] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c<br>
[2]PETSC ERROR: [2] User provided function line 160<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[INTERNAL ERROR: Invalid error class (66) encountered while returning from<br>
MPI_Allreduce. Please file a bug report. No error stack is available.<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_3]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
[4]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[4]PETSC ERROR: Caught signal number 3 Quit: Some other process (or<br>
the batch system) has told this process to end<br>
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[4]PETSC ERROR: or see<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[4]PETSC</a><br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to<br>
find memory corruption errors<br>
[4]PETSC ERROR: likely location of problem given in stack below<br>
[4]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[4]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[4]PETSC ERROR: is given.<br>
[4]PETSC ERROR: [4] MatAssemblyBegin_MPIAIJ line 462<br>
src/mat/impls/aij/mpi/mpiaij.c<br>
[4]PETSC ERROR: [4] MatAssemblyBegin line 4553 src/mat/interface/matrix.c<br>
[4]PETSC ERROR: [4] User provided functiINTERNAL ERROR: Invalid error<br>
class (66) encountered while returning from<br>
MPI_Allreduce. Please file a bug report. No error stack is available.<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_5]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
[6]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[6]PETSC ERROR: Caught signal number 3 Quit: Some other process (or<br>
the batch system) has told this process to end<br>
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[6]PETSC ERROR: or see<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[6]PETSC</a><br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to<br>
find memory corruption errors<br>
[6]PETSC ERROR: likely location of problem given in stack below<br>
[6]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[6]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[6]PETSC ERROR: is given.<br>
[6]PETSC ERROR: [6] MatAssemblyBegin_MPIAIJ line 462<br>
src/mat/impls/aij/mpi/mpiaij.c<br>
[6]PETSC ERROR: [6] MatAssemblyBegin line 4553 src/mat/interface/matrix.c<br>
[6]PETSC ERROR: [6] User provided functiINTERNAL ERROR: Invalid error<br>
class (66) encountered while returning from<br>
MPI_Allreduce. Please file a bug report. No error stack is available.<br>
Fatal error in MPI_Allreduce: Error message texts are not<br>
available[cli_8]: aborting job:<br>
Fatal error in MPI_Allreduce: Error message texts are not available<br>
[10]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[10]PETSC ERROR: Caught signal number 3 Quit: Some other process (or<br>
the batch system) has told this process to end<br>
[10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[10]PETSC ERROR: or see<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[10]PETSC</a><br>
ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to<br>
find memory corruption errors<br>
[10]PETSC ERROR: likely location of problem given in stack below<br>
[10]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[10]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[10]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[10]PETSC ERROR: is given.<br>
[10]PETSC ERROR: [10] VecAssemblyBegin line 157 src/vec/vec/interface/vector.c<br>
[10]PETSC ERROR: [10] User provided function line 160<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/on line 294<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[0]PETSC ERROR: [0] User provided function line 627<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[0]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[0]PETSC ERROR: Signal received!<br>
[0]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[0]PETSC ERROR: See docs/index.html for manual pages.<br>
[0]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[0]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug<br>
6 00:35:58 2011<br>
[0]PETSC ERROR: Libraries linked from<br>
/home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib<br>
[0]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011<br>
[0]PETSC ERROR: Config2]PETSC ERROR: [2] User provided function line<br>
294 "unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[2]PETSC ERROR: [2] User provided function line 627<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[2]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[2]PETSC ERROR: Signal received!<br>
[2]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[2]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[2]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[2]PETSC ERROR: See docs/index.html for manual pages.<br>
[2]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[2]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug<br>
6 00:35:58 2011<br>
[2]PETSC ERROR: Libraries linked from<br>
/home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib<br>
[2]PETSC ERROR: Configure run at Sat Aug on line 294<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[4]PETSC ERROR: [4] User provided function line 627<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[4]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[4]PETSC ERROR: Signal received!<br>
[4]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[4]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[4]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[4]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[4]PETSC ERROR: See docs/index.html for manual pages.<br>
[4]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[4]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug<br>
6 00:35:58 2011<br>
[4]PETSC ERROR: Libraries linked from<br>
/home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib<br>
[4]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011<br>
[4]PETSC ERROR: Configon line 294<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[6]PETSC ERROR: [6] User provided function line 627<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[6]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[6]PETSC ERROR: Signal received!<br>
[6]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[6]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[6]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[6]PETSC ERROR: See docs/index.html for manual pages.<br>
[6]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[6]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug<br>
6 00:35:58 2011<br>
[6]PETSC ERROR: Libraries linked from<br>
/home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib<br>
[6]PETSC ERROR: Configure run at Sat Aug 6 00:02:58 2011<br>
[6]PETSC ERROR: ConfigSM3T4mpi.cxx<br>
[10]PETSC ERROR: [10] User provided function line 294<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[10]PETSC ERROR: [10] User provided function line 627<br>
"unknowndirectory/"/home/dsz/src/framework/trunk/solve/SM3T4mpi.cxx<br>
[10]PETSC ERROR: --------------------- Error Message<br>
------------------------------------<br>
[10]PETSC ERROR: Signal received!<br>
[10]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[10]PETSC ERROR: Petsc Release Version 3.1.0, Patch 8, Thu Mar 17<br>
13:37:48 CDT 2011<br>
[10]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[10]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[10]PETSC ERROR: See docs/index.html for manual pages.<br>
[10]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[10]PETSC ERROR: Unknown Name on a linux-gnu named nexo by dsz Sat Aug<br>
6 00:35:58 2011<br>
[10]PETSC ERROR: Libraries linked from<br>
/home/dsz/pack/petsc-3.1-p8/linux-gnu-c-debug/lib<br>
[10]PETSC ERRure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8<br>
PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1<br>
--download-mpich=1 --download-hypre=1 --with-parmetis=1<br>
--download-parmetis=1 --with-x=0 --with-debugging=1<br>
[0]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[0]PETSC ERROR: User provided function() line 0 in unknown directory<br>
unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0[cli_0]:<br>
aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
6 00:02:58 2011<br>
[2]PETSC ERROR: Configure options<br>
PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug<br>
--download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1<br>
--with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1<br>
[2]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[2]PETSC ERROR: User provided function() line 0 in unknown directory<br>
unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2[cli_2]:<br>
aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 2<br>
ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8<br>
PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1<br>
--download-mpich=1 --download-hypre=1 --with-parmetis=1<br>
--download-parmetis=1 --with-x=0 --with-debugging=1<br>
[4]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[4]PETSC ERROR: User provided function() line 0 in unknown directory<br>
unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4[cli_4]:<br>
aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4<br>
ure options PETSC_DIR=/home/dsz/pack/petsc-3.1-p8<br>
PETSC_ARCH=linux-gnu-c-debug --download-f-blas-lapack=1<br>
--download-mpich=1 --download-hypre=1 --with-parmetis=1<br>
--download-parmetis=1 --with-x=0 --with-debugging=1<br>
[6]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[6]PETSC ERROR: User provided function() line 0 in unknown directory<br>
unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6[cli_6]:<br>
aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 6<br>
OR: Configure run at Sat Aug 6 00:02:58 2011<br>
[10]PETSC ERROR: Configure options<br>
PETSC_DIR=/home/dsz/pack/petsc-3.1-p8 PETSC_ARCH=linux-gnu-c-debug<br>
--download-f-blas-lapack=1 --download-mpich=1 --download-hypre=1<br>
--with-parmetis=1 --download-parmetis=1 --with-x=0 --with-debugging=1<br>
[10]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[10]PETSC ERROR: User provided function() line 0 in unknown directory<br>
unknown file<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10[cli_10]:<br>
aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10<br>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>