You can get a stack trace using the debugger. We also recommend tracking this<div>down using valgrind.</div><div><br></div><div> Matt<br><br><div class="gmail_quote">On Thu, Apr 15, 2010 at 9:14 PM, Harsh Sharma <span dir="ltr"><<a href="mailto:hsharma.tgjobs@gmail.com">hsharma.tgjobs@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi,<br><br>I'm a first-time user of the PETSc toolkit.<br><br>I'm getting a "Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range" error.<br>
<br>My program (called "pPCA_makeC0" below) is doing simple stuff : create a bunch of vectors, set its components to random numbers using VecSetRandom(), then compute the NORM_2 type vector-norm for each of them, and finally write them to a binary file.<br>
<br>My vectors are quite large (2821728 dimensions) and even if I create just one such vector, the above-mentioned error occurs. From the output of the program, it appears that MPI is having some issue with the binary-file-writing part of the program.<br>
<br>This problem occurs regardless of the number of processes/processors I use when invoking petscmpiexec. I've pasted the erroneous output at the end of this mail, for two scenarios: 1 processor and 1 or 5 processes.<br>
<br>Any help in resolving this would be much appreciated. Thanks!<br><br>Best,<br>Harsh<br><br>-------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
output of my program (D = vector-dimension = 2821728 here, k = number of vectors = 5 here), 1 process, 1 processor<br>-------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
<br>/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec -np 1 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5<br>2-norm of column 1 of C0 = 969.272<br>
2-norm of column 2 of C0 = 969.218<br>2-norm of column 3 of C0 = 969.087<br>2-norm of column 4 of C0 = 969.599<br>2-norm of column 5 of C0 = 969.547<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on linux or man libgmalloc on Apple to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>[0]PETSC ERROR: is given.<br>[0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c<br>
[0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[0]PETSC ERROR: Signal received!<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[0]PETSC ERROR: See docs/index.html for manual pages.<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:03:44 2010<br>
[0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>[0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0<br>
rank 0 in job 3 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks<br> exit status of rank 0: return code 59<br><br><br>
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
output of my program (D = vector-dimension = 2821728 here, k = number of
vectors = 5 here), 5 processes, 1 processor<br>
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br><br><br>/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec -np 5 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5<br>
2-norm of column 1 of C0 = 970.149<br>2-norm of column 2 of C0 = 969.699<br>2-norm of column 3 of C0 = 969.517<br>2-norm of column 4 of C0 = 970.253<br>2-norm of column 5 of C0 = 969.81<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on linux or man libgmalloc on Apple to find memory corruption errors<br>
[0]PETSC ERROR: likely location of problem given in stack below<br>[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>[1]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[1]PETSC ERROR: Write to file failed!<br>[1]PETSC ERROR: Error writing to file.!<br>[1]PETSC ERROR: ------------------------------------------------------------------------<br>[1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>
[1]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[1]PETSC ERROR: See docs/index.html for manual pages.<br>[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:09:23 2010<br>[1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>
[1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cwo[2]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[2]PETSC ERROR: Write to file failed!<br>[2]PETSC ERROR: Error writing to file.!<br>[2]PETSC ERROR: ------------------------------------------------------------------------<br>[2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>
[2]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[2]PETSC ERROR: See docs/index.html for manual pages.<br>[2]PETSC ERROR: ------------------------------------------------------------------------<br>
[2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:09:23 2010<br>[2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>
[2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>[1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c<br>[1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/fileio/sysio.c<br>
[1]PETSC ERROR: --------------------- Error Message ------------------------------------<br>[1]PETSC ERROR: Error: Unable to write C0 row dimension D!<br>[1]PETSC ERROR: ------------------------------------------------------------------------<br>
[1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>[1]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[1]PETSC ERROR: See docs/index.html forkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[2]PETSC ERROR: ------------------------------------------------------------------------<br>[2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c<br>[2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/fileio/sysio.c<br>
[2]PETSC ERROR: --------------------- Error Message ------------------------------------<br>[2]PETSC ERROR: Error: Unable to write C0 row dimension D!<br>[2]PETSC ERROR: ------------------------------------------------------------------------<br>
[2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>[2]PETSC ERROR: See docs/changes/index.html for recent updates.<br>[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
[2]PETSC ERROR: See docs/index.html for manual pages.<br>[1]PETSC ERROR: ------------------------------------------------------------------------<br>[1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:09:23 2010<br>
[1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>[1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[1]PETSC ERROR: ------------------------------------------------------------------------<br>[1]PETSC ERROR: User provided function() line 123 in /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c<br>r manual pages.<br>
[2]PETSC ERROR: ------------------------------------------------------------------------<br>[2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:09:23 2010<br>
[2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>[2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[2]PETSC ERROR: ------------------------------------------------------------------------<br>[2]PETSC ERROR: User provided function() line 123 in /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c<br>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[0]PETSC ERROR: INSTEAD the line number of the start of the function<br>[0]PETSC ERROR: is given.<br>[0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c<br>
[0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/fileio/sysio.c<br>[0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
[0]PETSC ERROR: Signal received!<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1 11:01:51 CST 2010<br>[0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>[0]PETSC ERROR: See docs/index.html for manual pages.<br>[0]PETSC ERROR: ------------------------------------------------------------------------<br>
[0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named <a href="http://ifp-32.ifp.uiuc.edu" target="_blank">ifp-32.ifp.uiuc.edu</a> by hsharma Thu Apr 15 21:09:23 2010<br>
[0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib<br>[0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010<br>[0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --with-scalar-type=real --with-precision=single --with-c++-support --with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1<br>
[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file<br>[3]PETSC ERROR: rank 2 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks<br>
exit status of rank 2: return code 1<br>rank 1 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort of all ranks<br> exit status of rank 1: return code 1<br><br>
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
</blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<br>
</div>