[petsc-users] Re-initializing petsc again
Suraj Prabhakaran
suraj.prabhakaran at gmail.com
Tue Sep 3 13:58:18 CDT 2013
Hello,
Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows.
main()
{
PetscInitialize()
...
...
AOCreate()..
AO operations..
AODestroy()..
...
...
PetscFinalize()
PETSC_COMM_WORLD = A new communicator (which includes more processes)
PetscInitialize()
...
...
AOCreate()..
AO operations..
AODestroy()
...
...
PetscFinalize()
}
So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem!
Best,
Suraj
-------------------------ERROR starts here-------------------------------
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR: INSTEAD the line number of the start of the function
[0]PETSC ERROR: is given.
[0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c
[0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c
[0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c
[0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c
[0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c
[0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c
[0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c
[0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR: INSTEAD the line number of the start of the function
[1]PETSC ERROR: is given.
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Signal received!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[2]PETSC ERROR: likely location of problem given in stack below
[2]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[2]PETSC ERROR: INSTEAD the line number of the start of the function
[2]PETSC ERROR: is given.
[2]PETSC ERROR: --------------------- Error Message ------------------------------------
[2]PETSC ERROR: Signal received!
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[3]PETSC ERROR: likely location of problem given in stack below
[3]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[3]PETSC ERROR: INSTEAD the line number of the start of the function
[3]PETSC ERROR: is given.
[3]PETSC ERROR: --------------------- Error Message ------------------------------------
[3]PETSC ERROR: Signal received!
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[3]PETSC ERROR: See docs/changes/index.html for recent updates.
[3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[3]PETSC ERROR: See docs/index.html for manual pages.
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[4]PETSC ERROR: likely location of problem given in stack below
[4]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[4]PETSC ERROR: INSTEAD the line number of the start of the function
[4]PETSC ERROR: is given.
[4]PETSC ERROR: --------------------- Error Message ------------------------------------
[4]PETSC ERROR: Signal received!
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[4]PETSC ERROR: See docs/changes/index.html for recent updates.
[4]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[4]PETSC ERROR: See docs/index.html for manual pages.
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[5]PETSC ERROR: likely location of problem given in stack below
[5]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[5]PETSC ERROR: INSTEAD the line number of the start of the function
[5]PETSC ERROR: is given.
[5]PETSC ERROR: --------------------- Error Message ------------------------------------
[5]PETSC ERROR: Signal received!
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[5]PETSC ERROR: See docs/changes/index.html for recent updates.
[5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[5]PETSC ERROR: See docs/index.html for manual pages.
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[6]PETSC ERROR: likely location of problem given in stack below
[6]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[6]PETSC ERROR: INSTEAD the line number of the start of the function
[6]PETSC ERROR: is given.
[6]PETSC ERROR: --------------------- Error Message ------------------------------------
[6]PETSC ERROR: Signal received!
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[6]PETSC ERROR: See docs/changes/index.html for recent updates.
[6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[6]PETSC ERROR: See docs/index.html for manual pages.
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
[7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[7]PETSC ERROR: likely location of problem given in stack below
[7]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[7]PETSC ERROR: INSTEAD the line number of the start of the function
[7]PETSC ERROR: is given.
[7]PETSC ERROR: --------------------- Error Message ------------------------------------
[7]PETSC ERROR: Signal received!
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
[7]PETSC ERROR: See docs/changes/index.html for recent updates.
[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[7]PETSC ERROR: See docs/index.html for manual pages.
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
[7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
[7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
[7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
--------------------------------------------------------------------------
mpiexec has exited due to process rank 0 with PID 2670 on
node grsacc17 exiting improperly. There are two reasons this could occur:
1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.
2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"
This may have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).
More information about the petsc-users
mailing list