[petsc-users] PetscBinaryWrite/PetscBinarySynchronizedWrite SEGV fault

Harsh Sharma hsharma.tgjobs at gmail.com
Thu Apr 15 21:14:49 CDT 2010 

Hi,

I'm a first-time user of the PETSc toolkit.

I'm getting a "Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range" error.

My program (called "pPCA_makeC0" below) is doing simple stuff : create a
bunch of vectors, set its components to random numbers using VecSetRandom(),
then compute the NORM_2 type vector-norm for each of them, and finally write
them to a binary file.

My vectors are quite large (2821728 dimensions) and even if I create just
one such vector, the above-mentioned error occurs. From the output of the
program, it appears that MPI is having some issue with the
binary-file-writing part of the program.

This problem occurs regardless of the number of processes/processors I use
when invoking petscmpiexec. I've pasted the erroneous output at the end of
this mail, for two scenarios: 1 processor and 1 or 5 processes.

Any help in resolving this would be much appreciated. Thanks!

Best,
Harsh

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
output of my program (D = vector-dimension = 2821728 here, k = number of
vectors = 5 here), 1 process, 1 processor
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec
-np 1 ./pPCA_makeC0 -c 1 -m
/cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
2-norm of column 1 of C0 = 969.272
2-norm of column 2 of C0 = 969.218
2-norm of column 3 of C0 = 969.087
2-norm of column 4 of C0 = 969.599
2-norm of column 5 of C0 = 969.547
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR:
or try
http://valgrind.org on linux or man libgmalloc on Apple to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585
src/sys/fileio/sysio.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:03:44 2010
[0]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[0]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown
file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
rank 0 in job 3  ifp-32.ifp.uiuc.edu_57355   caused collective abort of all
ranks
  exit status of rank 0: return code 59


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
output of my program (D = vector-dimension = 2821728 here, k = number of
vectors = 5 here), 5 processes, 1 processor
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------


/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec
-np 5 ./pPCA_makeC0 -c 1 -m
/cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
2-norm of column 1 of C0 = 970.149
2-norm of column 2 of C0 = 969.699
2-norm of column 3 of C0 = 969.517
2-norm of column 4 of C0 = 970.253
2-norm of column 5 of C0 = 969.81
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR:
or try
http://valgrind.org on linux or man libgmalloc on Apple to find memory
corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Write to file failed!
[1]PETSC ERROR: Error writing to file.!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:09:23 2010
[1]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[1]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cwo[2]PETSC ERROR: --------------------- Error
Message ------------------------------------
[2]PETSC ERROR: Write to file failed!
[2]PETSC ERROR: Error writing to file.!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:09:23 2010
[2]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[2]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
[1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in
src/sys/fileio/sysio.c
[1]PETSC ERROR: --------------------- Error Message
------------------------------------
[1]PETSC ERROR: Error: Unable to write C0 row dimension D!
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html
forkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
[2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in
src/sys/fileio/sysio.c
[2]PETSC ERROR: --------------------- Error Message
------------------------------------
[2]PETSC ERROR: Error: Unable to write C0 row dimension D!
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:09:23 2010
[1]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[1]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: User provided function() line 123 in
/cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
r manual pages.
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:09:23 2010
[2]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[2]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[2]PETSC ERROR:
------------------------------------------------------------------------
[2]PETSC ERROR: User provided function() line 123 in
/cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
[0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585
src/sys/fileio/sysio.c
[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
CST 2010
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
hsharma Thu Apr 15 21:09:23 2010
[0]PETSC ERROR: Libraries linked from
/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
[0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
[0]PETSC ERROR: Configure options
--prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
--with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
--with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
--with-large-file-io=1 --with-shared=0 --with-scalar-type=real
--with-precision=single --with-c++-support --with-c-support
--with-64-bit-indices=0 --with-log=1 --with-info=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown
file
[3]PETSC ERROR: rank 2 in job 4  ifp-32.ifp.uiuc.edu_57355   caused
collective abort of all ranks
  exit status of rank 2: return code 1
rank 1 in job 4  ifp-32.ifp.uiuc.edu_57355   caused collective abort of all
ranks
  exit status of rank 1: return code 1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20100415/d7dcc518/attachment.htm>

More information about the petsc-users mailing list