[petsc-users] Error on process numbers greater than 2

Thu Aug 23 06:34:59 CDT 2012

On Thu, Aug 23, 2012 at 5:25 AM, Arun Kamath <arun.kamath.m at gmail.com>wrote:

> Hi,
> I am a new user and was running the example problems.
>
> The given example in the package, ex2
> (src/ksp/ksp/examples/tutorials/ex2.c), works fine with two processors
> (mpiexec -n 2 ex2) but fails to work with any number greater than 2. For
> example:
> mpiexec -np 3 ex2, returns :
>
> [1]PETSC ERROR: PetscTrFreeDefault() called from
> PetscGatherMessageLengths() line 140 in src/sys/utils/mpimesg.c
> [1]PETSC ERROR: Block [id=0(80)] at address 0x7f93799a1a20 is corrupted
> (probably write past end of array)
> [1]PETSC ERROR: Block allocated in PetscGatherMessageLengths() line 115 in
> src/sys/utils/mpimesg.c
> [1]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [1]PETSC ERROR: Memory corruption!
> [1]PETSC ERROR: Corrupted memory!
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00
> CDT 2012
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs
> Thu Aug 23 12:21:58 2012
> [1]PETSC ERROR: Libraries linked from
> /Users/bihs/petsc/arch-darwin-c-debug/lib
> [1]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012
> [1]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/local/include/openmpi --with-fc=0
> --download-f2cblaslapack
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: PetscTrFreeDefault() line 299 in src/sys/memory/mtr.c
> [1]PETSC ERROR: PetscGatherMessageLengths() line 140 in
> src/sys/utils/mpimesg.c
> [1]PETSC ERROR: VecScatterCreate_PtoS() line 1664 in
> src/vec/vec/utils/vpscat.c
> [1]PETSC ERROR: VecScatterCreate() line 1331 in src/vec/vec/utils/vscat.c
> [1]PETSC ERROR: MatSetUpMultiply_MPIAIJ() line 150 in
> src/mat/impls/aij/mpi/mmaij.c
> [1]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 680 in
> src/mat/impls/aij/mpi/mpiaij.c
> [1]PETSC ERROR: MatAssemblyEnd() line 4866 in src/mat/interface/matrix.c
> [1]PETSC ERROR: main() line 107 in src/ksp/ksp/examples/tutorials/ex2.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
> with errorcode 1.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] VecScatterCreate_PtoS line 1631
> src/vec/vec/utils/vpscat.c
> [0]PETSC ERROR: [0] VecScatterCreate line 844 src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: [0] MatSetUpMultiply_MPIAIJ line 26
> src/mat/impls/aij/mpi/mmaij.c
> [0]PETSC ERROR: [0] MatAssemblyEnd_MPIAIJ line 646
> src/mat/impls/aij/mpi/mpiaij.c
> [0]PETSC ERROR: [0] MatAssemblyEnd line 4857 src/mat/interface/matrix.c
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00
> CDT 2012
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs
> Thu Aug 23 12:21:58 2012
> [0]PETSC ERROR: Libraries linked from
> /Users/bihs/petsc/arch-darwin-c-debug/lib
> [0]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012
> [0]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/local/include/openmpi --with-fc=0
> --download-f2cblaslapack
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the
> batch system) has told this process to end
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSCERROR: or try
> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory
> corruption errors
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [2]PETSC ERROR:       INSTEAD the line number of the start of the function
> [2]PETSC ERROR:       is given.
> [2]PETSC ERROR: [2] VecScatterCreate_PtoS line 1631
> src/vec/vec/utils/vpscat.c
> [2]PETSC ERROR: [2] VecScatterCreate line 844 src/vec/vec/utils/vscat.c
> [2]PETSC ERROR: [2] MatSetUpMultiply_MPIAIJ line 26
> src/mat/impls/aij/mpi/mmaij.c
> [2]PETSC ERROR: [2] MatAssemblyEnd_MPIAIJ line 646
> src/mat/impls/aij/mpi/mpiaij.c
> [2]PETSC ERROR: [2] MatAssemblyEnd line 4857 src/mat/interface/matrix.c
> [2]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [2]PETSC ERROR: Signal received!
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 2, Fri Jul 13 15:42:00
> CDT 2012
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: ex2 on a arch-darw named ball-9120.ivt.ntnu.no by bihs
> Thu Aug 23 12:21:58 2012
> [2]PETSC ERROR: Libraries linked from
> /Users/bihs/petsc/arch-darwin-c-debug/lib
> [2]PETSC ERROR: Configure run at Tue Aug 21 15:07:34 2012
> [2]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/local/include/openmpi --with-fc=0
> --download-f2cblaslapack
> [2]PETSC ERROR:
> ------------------------------------------------------------------------
> [2]PETSC ERROR: User provided function() line 0 in unknown directory
> unknown file
>
> My machine has two 6 core Intel Xeon processors and have been running
> other programs that use MPI to run 12 processes.
> Could someone shed some light on why this is happening?
>

It looks like your MPI installation is bad. Try reconfiguring using
--download-mpich

   Matt

> Thanks.
>
> --
> Arun Kamath
> PhD Stipendiat
> NTNU, Trondheim
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120823/b0422562/attachment-0001.html>