[petsc-users] PetscBinaryWrite/PetscBinarySynchronizedWrite SEGV fault
Barry Smith
bsmith at mcs.anl.gov
Thu Apr 15 21:39:15 CDT 2010
You should run your code under valgrind http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#valgrind
to find the memory corruption or use of unitialized or out of range
memory that is causing the problem. With valgrind you will find
your bug in less then five minutes, without it you could waste hours
futzing around before finding the exact problem.
Barry
On Apr 15, 2010, at 9:14 PM, Harsh Sharma wrote:
> Hi,
>
> I'm a first-time user of the PETSc toolkit.
>
> I'm getting a "Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range" error.
>
> My program (called "pPCA_makeC0" below) is doing simple stuff :
> create a bunch of vectors, set its components to random numbers
> using VecSetRandom(), then compute the NORM_2 type vector-norm for
> each of them, and finally write them to a binary file.
>
> My vectors are quite large (2821728 dimensions) and even if I create
> just one such vector, the above-mentioned error occurs. From the
> output of the program, it appears that MPI is having some issue with
> the binary-file-writing part of the program.
>
> This problem occurs regardless of the number of processes/processors
> I use when invoking petscmpiexec. I've pasted the erroneous output
> at the end of this mail, for two scenarios: 1 processor and 1 or 5
> processes.
>
> Any help in resolving this would be much appreciated. Thanks!
>
> Best,
> Harsh
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> output of my program (D = vector-dimension = 2821728 here, k =
> number of vectors = 5 here), 1 process, 1 processor
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/
> bin/petscmpiexec -np 1 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/
> hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
> 2-norm of column 1 of C0 = 969.272
> 2-norm of column 2 of C0 = 969.218
> 2-norm of column 3 of C0 = 969.087
> 2-norm of column 4 of C0 = 969.599
> 2-norm of column 5 of C0 = 969.547
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -
> on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal
> [0]PETSC ERROR: or try http://valgrind.org on linux or man
> libgmalloc on Apple to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: INSTEAD the line number of the start of the
> function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/
> sysio.c
> [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/
> fileio/sysio.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:03:44 2010
> [0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-
> large-file-io=1 --with-shared=0 --with-scalar-type=real --with-
> precision=single --with-c++-support --with-c-support --with-64-bit-
> indices=0 --with-log=1 --with-info=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> rank 0 in job 3 ifp-32.ifp.uiuc.edu_57355 caused collective abort
> of all ranks
> exit status of rank 0: return code 59
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> output of my program (D = vector-dimension = 2821728 here, k =
> number of vectors = 5 here), 5 processes, 1 processor
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/
> bin/petscmpiexec -np 5 ./pPCA_makeC0 -c 1 -m /cworkspace/ifp-32-2/
> hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
> 2-norm of column 1 of C0 = 970.149
> 2-norm of column 2 of C0 = 969.699
> 2-norm of column 3 of C0 = 969.517
> 2-norm of column 4 of C0 = 970.253
> 2-norm of column 5 of C0 = 969.81
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
> Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -
> on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal
> [0]PETSC ERROR: or try http://valgrind.org on linux or man
> libgmalloc on Apple to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: --------------------- Stack Frames
> ------------------------------------
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Write to file failed!
> [1]PETSC ERROR: Error writing to file.!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010
> [1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cwo[2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Write to file failed!
> [2]PETSC ERROR: Error writing to file.!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010
> [2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-
> large-file-io=1 --with-shared=0 --with-scalar-type=real --with-
> precision=single --with-c++-support --with-c-support --with-64-bit-
> indices=0 --with-log=1 --with-info=1
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
> [1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/
> fileio/sysio.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Error: Unable to write C0 row dimension D!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html forkspace/ifp-32-2/hasegawa/
> hsharma/apps/myLib --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/mpich2-install --with-large-file-io=1 --with-shared=0 --
> with-scalar-type=real --with-precision=single --with-c++-support --
> with-c-support --with-64-bit-indices=0 --with-log=1 --with-info=1
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
> [2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in src/sys/
> fileio/sysio.c
> [2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Error: Unable to write C0 row dimension D!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010
> [1]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [1]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-
> large-file-io=1 --with-shared=0 --with-scalar-type=real --with-
> precision=single --with-c++-support --with-c-support --with-64-bit-
> indices=0 --with-log=1 --with-info=1
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 123 in /cworkspace/
> ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
> r manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010
> [2]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [2]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-
> large-file-io=1 --with-shared=0 --with-scalar-type=real --with-
> precision=single --with-c++-support --with-c-support --with-64-bit-
> indices=0 --with-log=1 --with-info=1
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: User provided function() line 123 in /cworkspace/
> ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR: INSTEAD the line number of the start of the
> function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/
> sysio.c
> [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585 src/sys/
> fileio/sysio.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb 1
> 11:01:51 CST 2010
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named
> ifp-32.ifp.uiuc.edu by hsharma Thu Apr 15 21:09:23 2010
> [0]PETSC ERROR: Libraries linked from /cworkspace/ifp-32-2/hasegawa/
> hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [0]PETSC ERROR: Configure options --prefix=/cworkspace/ifp-32-2/
> hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal --with-blas-lapack-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib --with-mpi-dir=/
> cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install --with-
> large-file-io=1 --with-shared=0 --with-scalar-type=real --with-
> precision=single --with-c++-support --with-c-support --with-64-bit-
> indices=0 --with-log=1 --with-info=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [3]PETSC ERROR: rank 2 in job 4 ifp-32.ifp.uiuc.edu_57355 caused
> collective abort of all ranks
> exit status of rank 2: return code 1
> rank 1 in job 4 ifp-32.ifp.uiuc.edu_57355 caused collective abort
> of all ranks
> exit status of rank 1: return code 1
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20100415/a8e533e6/attachment.htm>
More information about the petsc-users
mailing list