[petsc-dev] Installation crash on test

Thu Dec 12 14:24:30 CST 2013

I can't reproduce with current petsc-dev.

I'll suggest doing a clean build and see if the problem persists. If so
- try running a debugger to get the stack trace [with COPTFLAGS="-g -O2"]

git clean -f -d -x

Satish

On Thu, 12 Dec 2013, Satish Balay wrote:

> Ok - perhaps the attached configure.log didn't correspond to this
> build.
> 
> Will check with these options and see if I can reproduce.
> 
> Satish
> 
> On Thu, 12 Dec 2013, John Mousel wrote:
> 
> > Satish,
> > 
> > The script that pulled petsc-dev uses:
> > 
> > export PETSC_DIR=/home/augspurger/NumericalLibraries/petsc
> > git clone https://bitbucket.org/petsc/petsc
> > cd petsc
> > export PETSC_ARCH=gnu-opt
> > ./config/configure.py --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
> > --download-openmpi --download-metis --download-parmetis
> > --download-f-blas-lapack --download-cmake=1 --download-scalapack
> > --download-mumps --download-superlu_dist -with-debugging=0 COPTFLAGS="-O2"
> > FOPTFLAGS="-O2"
> > make all
> > make test
> > 
> > This is somehow not pulling with git?
> > 
> > 
> > John
> > 
> > 
> > On Thu, Dec 12, 2013 at 1:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > > [posting compressed logs would bypass the moderation on this list. Or
> > > one can send logs to petsc-maint]
> > >
> > >
> > > ================================================================================
> > > TEST configureInstallationMethod from
> > > PETSc.utilities.petscdir(/home/jmousel/NumericalLibraries/petsc-hg/petsc-dev/config/PETSc/utilities/petscdir.py:82)
> > > TESTING: configureInstallationMethod from
> > > PETSc.utilities.petscdir(config/PETSc/utilities/petscdir.py:82)
> > >       This is a Mercurial clone
> > >         Defined "VERSION_HG" to
> > > ""ae1852fa137384c3044d6feae2ef8a7da2eceaf8""
> > >         Defined "VERSION_DATE_HG" to ""Tue Oct 09 18:04:34 2012 -0500""
> > >         Defined "VERSION_BS_HG" to
> > > ""1a63457bc7b1d283c8380ea48550851ce1bc3091""
> > >         Defined "VERSION_BS_DATE_HG" to ""Mon Oct 08 16:33:25 2012 -0500""
> > >
> > > ================================================================================
> > >
> > >
> > > This petsc-dev snapshot is a 'mercurial clone' - and is more than a year
> > > old.
> > >
> > > Could you try using latest petsc-dev [via git]?
> > >
> > > http://www.mcs.anl.gov/petsc/developers/index.html#obtaining
> > >
> > > thanks,
> > > Satish
> > >
> > > On Thu, 12 Dec 2013, Augspurger, Mike wrote:
> > >
> > > > I just pulled and installed petsc-dev and bombed the tests with the error
> > > > message below. I've attached configure.log.
> > > >
> > > > Now to check if the libraries are working do:
> > > > make PETSC_DIR=/home/augspurger/NumericalLibraries/petsc
> > > PETSC_ARCH=gnu-opt
> > > > test
> > > > =========================================
> > > > Running test examples to verify correct installation
> > > > Using PETSC_DIR=/home/augspurger/NumericalLibraries/petsc and
> > > > PETSC_ARCH=gnu-opt
> > > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI
> > > > process
> > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > > > probably memory access out of range
> > > > [0]PETSC ERROR: Try option -start_in_debugger or
> > > -on_error_attach_debugger
> > > > [0]PETSC ERROR: or see
> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR:
> > > > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> > > memory
> > > > corruption errors
> > > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > > and
> > > > run
> > > > [0]PETSC ERROR: to get more information on the crash.
> > > > [0]PETSC ERROR: --------------------- Error Message
> > > > ------------------------------------
> > > > [0]PETSC ERROR: Signal received!
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: Petsc Development GIT revision:
> > > > 58e0e1d1cef173c8daf844a24cdc569afb7a2817  GIT Date: 2013-12-11 15:53:51
> > > > -0600
> > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > > > [0]PETSC ERROR: See docs/index.html for manual pages.
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: ./ex19 on a gnu-opt named lan.iihr.uiowa.edu by
> > > augspurger
> > > > Thu Dec 12 10:48:23 2013
> > > > [0]PETSC ERROR: Libraries linked from
> > > > /home/augspurger/NumericalLibraries/petsc/gnu-opt/lib
> > > > [0]PETSC ERROR: Configure run at Thu Dec 12 10:46:20 2013
> > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> > > > --with-fc=gfortran --download-openmpi --download-metis
> > > --download-parmetis
> > > > --download-f-blas-lapack --download-cmake=1 --download-scalapack
> > > > --download-mumps --download-superlu_dist -with-debugging=0 COPTFLAGS=-O2
> > > > FOPTFLAGS=-O2
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: User provided function() line 0 in  unknown file
> > > >
> > > --------------------------------------------------------------------------
> > > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > > with errorcode 59.
> > > >
> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > > You may or may not see output from other processes, depending on
> > > > exactly when Open MPI kills them.
> > > >
> > > --------------------------------------------------------------------------
> > > >
> > > --------------------------------------------------------------------------
> > > > mpiexec has exited due to process rank 0 with PID 26691 on
> > > > node lan.iihr.uiowa.edu exiting improperly. There are two reasons this
> > > > could occur:
> > > >
> > > > 1. this process did not call "init" before exiting, but others in
> > > > the job did. This can cause a job to hang indefinitely while it waits
> > > > for all processes to call "init". By rule, if one process calls "init",
> > > > then ALL processes must call "init" prior to termination.
> > > >
> > > > 2. this process called "init", but exited without calling "finalize".
> > > > By rule, all processes that call "init" MUST call "finalize" prior to
> > > > exiting or it will be considered an "abnormal termination"
> > > >
> > > > This may have caused other processes in the application to be
> > > > terminated by signals sent by mpiexec (as reported here).
> > > >
> > > --------------------------------------------------------------------------
> > > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI
> > > > processes
> > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > > > lid velocity = 0.0016, prandtl # = 1, grashof # = 1
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > > > probably memory access out of range
> > > > [0]PETSC ERROR: Try option -start_in_debugger or
> > > -on_error_attach_debugger
> > > > [1]PETSC ERROR: [0]PETSC ERROR: or see
> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSCERROR:
> > > > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> > > memory
> > > > corruption errors
> > > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > > and
> > > > run
> > > > [0]PETSC ERROR: to get more information on the crash.
> > > > [0]PETSC ERROR: --------------------- Error Message
> > > > ------------------------------------
> > > > [0]PETSC ERROR: Signal received!
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: Petsc Development GIT revision:
> > > > 58e0e1d1cef173c8daf844a24cdc569afb7a2817  GIT Date: 2013-12-11 15:53:51
> > > > -0600
> > > > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > > > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > > > [0]PETSC ERROR: See docs/index.html for manual pages.
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: ./ex19 on a gnu-opt named lan.iihr.uiowa.edu by
> > > augspurger
> > > > Thu Dec 12 10:48:24 2013
> > > > [0]PETSC ERROR: Libraries linked from
> > > > /home/augspurger/NumericalLibraries/petsc/gnu-opt/lib
> > > > [0]PETSC ERROR: Configure run at Thu Dec 12 10:46:20 2013
> > > > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> > > > --with-fc=gfortran --download-openmpi --download-metis
> > > --download-parmetis
> > > > --download-f-blas-lapack --download-cmake=1 --download-scalapack
> > > > --download-mumps --download-superlu_dist -with-debugging=0 COPTFLAGS=-O2
> > > > FOPTFLAGS=-O2
> > > > [0]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [0]PETSC ERROR: User provided function() line 0 in  unknown file
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> > > > probably memory access out of range
> > > >
> > > --------------------------------------------------------------------------
> > > > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> > > > with errorcode 59.
> > > >
> > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > > > You may or may not see output from other processes, depending on
> > > > exactly when Open MPI kills them.
> > > >
> > > --------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Try option -start_in_debugger or
> > > -on_error_attach_debugger
> > > > [1]PETSC ERROR: or see
> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR:
> > > > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> > > memory
> > > > corruption errors
> > > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > > and
> > > > run
> > > > [1]PETSC ERROR: to get more information on the crash.
> > > > [1]PETSC ERROR: --------------------- Error Message
> > > > ------------------------------------
> > > > [1]PETSC ERROR: Signal received!
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Petsc Development GIT revision:
> > > > 58e0e1d1cef173c8daf844a24cdc569afb7a2817  GIT Date: 2013-12-11 15:53:51
> > > > -0600
> > > > [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> > > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > > > [1]PETSC ERROR: See docs/index.html for manual pages.
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: ./ex19 on a gnu-opt named lan.iihr.uiowa.edu by
> > > augspurger
> > > > Thu Dec 12 10:48:24 2013
> > > > [1]PETSC ERROR: Libraries linked from
> > > > /home/augspurger/NumericalLibraries/petsc/gnu-opt/lib
> > > > [1]PETSC ERROR: Configure run at Thu Dec 12 10:46:20 2013
> > > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> > > > --with-fc=gfortran --download-openmpi --download-metis
> > > --download-parmetis
> > > > --download-f-blas-lapack --download-cmake=1 --download-scalapack
> > > > --download-mumps --download-superlu_dist -with-debugging=0 COPTFLAGS=-O2
> > > > FOPTFLAGS=-O2
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: User provided function() line 0 in  unknown file
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> > > > batch system) has told this process to end
> > > > [1]PETSC ERROR: Try option -start_in_debugger or
> > > -on_error_attach_debugger
> > > > [1]PETSC ERROR: or see
> > > > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSCERROR:
> > > > or try http://valgrind.org on GNU/linux and Apple Mac OS X to find
> > > memory
> > > > corruption errors
> > > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link,
> > > and
> > > > run
> > > > [1]PETSC ERROR: to get more information on the crash.
> > > > [1]PETSC ERROR: --------------------- Error Message
> > > > ------------------------------------
> > > > [1]PETSC ERROR: Signal received!
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: Petsc Development GIT revision:
> > > > 58e0e1d1cef173c8daf844a24cdc569afb7a2817  GIT Date: 2013-12-11 15:53:51
> > > > -0600
> > > > [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> > > > [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > > > [1]PETSC ERROR: See docs/index.html for manual pages.
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: ./ex19 on a gnu-opt named lan.iihr.uiowa.edu by
> > > augspurger
> > > > Thu Dec 12 10:48:24 2013
> > > > [1]PETSC ERROR: Libraries linked from
> > > > /home/augspurger/NumericalLibraries/petsc/gnu-opt/lib
> > > > [1]PETSC ERROR: Configure run at Thu Dec 12 10:46:20 2013
> > > > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> > > > --with-fc=gfortran --download-openmpi --download-metis
> > > --download-parmetis
> > > > --download-f-blas-lapack --download-cmake=1 --download-scalapack
> > > > --download-mumps --download-superlu_dist -with-debugging=0 COPTFLAGS=-O2
> > > > FOPTFLAGS=-O2
> > > > [1]PETSC ERROR:
> > > > ------------------------------------------------------------------------
> > > > [1]PETSC ERROR: User provided function() line 0 in  unknown file
> > > >
> > > --------------------------------------------------------------------------
> > > > mpiexec has exited due to process rank 0 with PID 26695 on
> > > > node lan.iihr.uiowa.edu exiting improperly. There are two reasons this
> > > > could occur:
> > > >
> > > > 1. this process did not call "init" before exiting, but others in
> > > > the job did. This can cause a job to hang indefinitely while it waits
> > > > for all processes to call "init". By rule, if one process calls "init",
> > > > then ALL processes must call "init" prior to termination.
> > > >
> > > > 2. this process called "init", but exited without calling "finalize".
> > > > By rule, all processes that call "init" MUST call "finalize" prior to
> > > > exiting or it will be considered an "abnormal termination"
> > > >
> > > > This may have caused other processes in the application to be
> > > > terminated by signals sent by mpiexec (as reported here).
> > > >
> > > --------------------------------------------------------------------------
> > > > [lan.iihr.uiowa.edu:26694] 1 more process has sent help message
> > > > help-mpi-api.txt / mpi-abort
> > > > [lan.iihr.uiowa.edu:26694] Set MCA parameter "orte_base_help_aggregate"
> > > to
> > > > 0 to see all help / error messages
> > > > Fortran example src/snes/examples/tutorials/ex5f run successfully with 1
> > > > MPI process
> > > > Completed test examples
> > > >
> > >
> > >
> > 
> 
> 