[petsc-users] PetscBinaryWrite/PetscBinarySynchronizedWrite SEGV fault

Thu Apr 15 21:31:31 CDT 2010

You can get a stack trace using the debugger. We also recommend tracking
this
down using valgrind.

  Matt

On Thu, Apr 15, 2010 at 9:14 PM, Harsh Sharma <hsharma.tgjobs at gmail.com>wrote:

> Hi,
>
> I'm a first-time user of the PETSc toolkit.
>
> I'm getting a "Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range" error.
>
> My program (called "pPCA_makeC0" below) is doing simple stuff : create a
> bunch of vectors, set its components to random numbers using VecSetRandom(),
> then compute the NORM_2 type vector-norm for each of them, and finally write
> them to a binary file.
>
> My vectors are quite large (2821728 dimensions) and even if I create just
> one such vector, the above-mentioned error occurs. From the output of the
> program, it appears that MPI is having some issue with the
> binary-file-writing part of the program.
>
> This problem occurs regardless of the number of processes/processors I use
> when invoking petscmpiexec. I've pasted the erroneous output at the end of
> this mail, for two scenarios: 1 processor and 1 or 5 processes.
>
> Any help in resolving this would be much appreciated. Thanks!
>
> Best,
> Harsh
>
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> output of my program (D = vector-dimension = 2821728 here, k = number of
> vectors = 5 here), 1 process, 1 processor
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec
> -np 1 ./pPCA_makeC0 -c 1 -m
> /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
> 2-norm of column 1 of C0 = 969.272
> 2-norm of column 2 of C0 = 969.218
> 2-norm of column 3 of C0 = 969.087
> 2-norm of column 4 of C0 = 969.599
> 2-norm of column 5 of C0 = 969.547
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try
> http://valgrind.org on linux or man libgmalloc on Apple to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585
> src/sys/fileio/sysio.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:03:44 2010
> [0]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [0]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> rank 0 in job 3  ifp-32.ifp.uiuc.edu_57355   caused collective abort of all
> ranks
>   exit status of rank 0: return code 59
>
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> output of my program (D = vector-dimension = 2821728 here, k = number of
> vectors = 5 here), 5 processes, 1 processor
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/bin/petscmpiexec
> -np 5 ./pPCA_makeC0 -c 1 -m
> /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/C0.mat -D 2821728 -k 5
> 2-norm of column 1 of C0 = 970.149
> 2-norm of column 2 of C0 = 969.699
> 2-norm of column 3 of C0 = 969.517
> 2-norm of column 4 of C0 = 970.253
> 2-norm of column 5 of C0 = 969.81
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSCERROR: or try
> http://valgrind.org on linux or man libgmalloc on Apple to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Write to file failed!
> [1]PETSC ERROR: Error writing to file.!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:09:23 2010
> [1]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [1]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cwo[2]PETSC ERROR: --------------------- Error
> Message ------------------------------------
> [2]PETSC ERROR: Write to file failed!
> [2]PETSC ERROR: Error writing to file.!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:09:23 2010
> [2]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [2]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
> [1]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in
> src/sys/fileio/sysio.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Error: Unable to write C0 row dimension D!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html
> forkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: PetscBinaryWrite() line 337 in src/sys/fileio/sysio.c
> [2]PETSC ERROR: PetscBinarySynchronizedWrite() line 588 in
> src/sys/fileio/sysio.c
> [2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Error: Unable to write C0 row dimension D!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:09:23 2010
> [1]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [1]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [1]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 123 in
> /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
> r manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:09:23 2010
> [2]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [2]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [2]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: User provided function() line 123 in
> /cworkspace/ifp-32-2/hasegawa/hsharma/testPPCA/pPCA_makeC0.c
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] PetscByteSwapScalar line 133 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscByteSwap line 179 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinaryWrite line 315 src/sys/fileio/sysio.c
> [0]PETSC ERROR: [0] PetscBinarySynchronizedWrite line 585
> src/sys/fileio/sysio.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 11, Mon Feb  1 11:01:51
> CST 2010
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./pPCA_makeC0 on a lx26-amd6 named ifp-32.ifp.uiuc.edu by
> hsharma Thu Apr 15 21:09:23 2010
> [0]PETSC ERROR: Libraries linked from
> /cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal/lib
> [0]PETSC ERROR: Configure run at Wed Mar 24 14:18:28 2010
> [0]PETSC ERROR: Configure options
> --prefix=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/petsc-3.0.0p11-fltReal
> --with-blas-lapack-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/myLib
> --with-mpi-dir=/cworkspace/ifp-32-2/hasegawa/hsharma/apps/mpich2-install
> --with-large-file-io=1 --with-shared=0 --with-scalar-type=real
> --with-precision=single --with-c++-support --with-c-support
> --with-64-bit-indices=0 --with-log=1 --with-info=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [3]PETSC ERROR: rank 2 in job 4  ifp-32.ifp.uiuc.edu_57355   caused
> collective abort of all ranks
>   exit status of rank 2: return code 1
> rank 1 in job 4  ifp-32.ifp.uiuc.edu_57355   caused collective abort of all
> ranks
>   exit status of rank 1: return code 1
>
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>

-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20100415/913c499d/attachment-0001.htm>