[petsc-users] Re-initializing petsc again
Barry Smith
bsmith at mcs.anl.gov
Tue Sep 3 14:01:41 CDT 2013
There is a problem with the 3.3 series when reinitializing PETSc. Can you switch to PETSc 3.4 which properly supports calling PetscInitialize() several times.
Barry
On Sep 3, 2013, at 1:58 PM, Suraj Prabhakaran <suraj.prabhakaran at gmail.com> wrote:
> Hello,
>
> Again, this relates to finalizing and initializing petsc in my application. I basically want to finalize petsc and initialize it again with a different MPI communicator. A rough sketch of my quite huge application is as follows.
>
>
> main()
> {
> PetscInitialize()
> ...
> ...
> AOCreate()..
> AO operations..
> AODestroy()..
> ...
> ...
> PetscFinalize()
> PETSC_COMM_WORLD = A new communicator (which includes more processes)
> PetscInitialize()
> ...
> ...
> AOCreate()..
> AO operations..
> AODestroy()
> ...
> ...
> PetscFinalize()
> }
>
> So after finalizing and initializing it back, I always get an error with one of the AO calls. Given below is the complete error I get when running with 8 procs. Can anyone give me a hint what the problem could be? I was told earlier, that finalizing and initializing petsc should not be a problem!
>
> Best,
> Suraj
>
> -------------------------ERROR starts here-------------------------------
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR: INSTEAD the line number of the start of the function
> [0]PETSC ERROR: is given.
> [0]PETSC ERROR: [0] PetscStrcmp line 414 /root/petsc/petsc-3.3-p7/src/sys/utils/str.c
> [0]PETSC ERROR: [0] PetscFListAdd line 202 /root/petsc/petsc-3.3-p7/src/sys/dll/reg.c
> [0]PETSC ERROR: [0] AORegister line 99 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoreg.c
> [0]PETSC ERROR: [0] AORegisterAll line 28 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/aoregall.c
> [0]PETSC ERROR: [0] AOInitializePackage line 49 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/dlregisdm.c
> [0]PETSC ERROR: [0] AOCreate line 530 /root/petsc/petsc-3.3-p7/src/dm/ao/interface/ao.c
> [0]PETSC ERROR: [0] AOCreateBasicIS line 381 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c
> [0]PETSC ERROR: [0] AOCreateBasic line 336 /root/petsc/petsc-3.3-p7/src/dm/ao/impls/basic/aobasic.c
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Signal received!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [0]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [0]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [0]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 59.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR: INSTEAD the line number of the start of the function
> [1]PETSC ERROR: is given.
> [1]PETSC ERROR: --------------------- Error Message ------------------------------------
> [1]PETSC ERROR: Signal received!
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [1]PETSC ERROR: See docs/changes/index.html for recent updates.
> [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [1]PETSC ERROR: See docs/index.html for manual pages.
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [1]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [1]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [1]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [2]PETSC ERROR: likely location of problem given in stack below
> [2]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [2]PETSC ERROR: INSTEAD the line number of the start of the function
> [2]PETSC ERROR: is given.
> [2]PETSC ERROR: --------------------- Error Message ------------------------------------
> [2]PETSC ERROR: Signal received!
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [2]PETSC ERROR: See docs/changes/index.html for recent updates.
> [2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [2]PETSC ERROR: See docs/index.html for manual pages.
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [2]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [2]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [2]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [2]PETSC ERROR: ------------------------------------------------------------------------
> [2]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [3]PETSC ERROR: likely location of problem given in stack below
> [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [3]PETSC ERROR: INSTEAD the line number of the start of the function
> [3]PETSC ERROR: is given.
> [3]PETSC ERROR: --------------------- Error Message ------------------------------------
> [3]PETSC ERROR: Signal received!
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [3]PETSC ERROR: See docs/changes/index.html for recent updates.
> [3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [3]PETSC ERROR: See docs/index.html for manual pages.
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [3]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [3]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [3]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [3]PETSC ERROR: ------------------------------------------------------------------------
> [3]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [4]PETSC ERROR: likely location of problem given in stack below
> [4]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [4]PETSC ERROR: INSTEAD the line number of the start of the function
> [4]PETSC ERROR: is given.
> [4]PETSC ERROR: --------------------- Error Message ------------------------------------
> [4]PETSC ERROR: Signal received!
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [4]PETSC ERROR: See docs/changes/index.html for recent updates.
> [4]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [4]PETSC ERROR: See docs/index.html for manual pages.
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [4]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [4]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [4]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [4]PETSC ERROR: ------------------------------------------------------------------------
> [4]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[5]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [5]PETSC ERROR: likely location of problem given in stack below
> [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [5]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [5]PETSC ERROR: INSTEAD the line number of the start of the function
> [5]PETSC ERROR: is given.
> [5]PETSC ERROR: --------------------- Error Message ------------------------------------
> [5]PETSC ERROR: Signal received!
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [5]PETSC ERROR: See docs/changes/index.html for recent updates.
> [5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [5]PETSC ERROR: See docs/index.html for manual pages.
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [5]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [5]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [5]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [5]PETSC ERROR: ------------------------------------------------------------------------
> [5]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [6]PETSC ERROR: likely location of problem given in stack below
> [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [6]PETSC ERROR: INSTEAD the line number of the start of the function
> [6]PETSC ERROR: is given.
> [6]PETSC ERROR: --------------------- Error Message ------------------------------------
> [6]PETSC ERROR: Signal received!
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [6]PETSC ERROR: See docs/changes/index.html for recent updates.
> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [6]PETSC ERROR: See docs/index.html for manual pages.
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [6]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [6]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [6]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> [7]PETSC ERROR: ------------------------------------------------------------------------
> [7]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end
> [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[7]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [7]PETSC ERROR: likely location of problem given in stack below
> [7]PETSC ERROR: --------------------- Stack Frames ------------------------------------
> [7]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [7]PETSC ERROR: INSTEAD the line number of the start of the function
> [7]PETSC ERROR: is given.
> [7]PETSC ERROR: --------------------- Error Message ------------------------------------
> [7]PETSC ERROR: Signal received!
> [7]PETSC ERROR: ------------------------------------------------------------------------
> [7]PETSC ERROR: Petsc Release Version 3.3.0, Patch 7, Sat May 11 22:15:24 CDT 2013
> [7]PETSC ERROR: See docs/changes/index.html for recent updates.
> [7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [7]PETSC ERROR: See docs/index.html for manual pages.
> [7]PETSC ERROR: ------------------------------------------------------------------------
> [7]PETSC ERROR: ./quadflow on a arch-linu named grsacc17 by grsprabh Tue Sep 3 20:56:21 2013
> [7]PETSC ERROR: Libraries linked from /opt/petsc3.3-1/lib
> [7]PETSC ERROR: Configure run at Mon Aug 19 13:56:50 2013
> [7]PETSC ERROR: Configure options --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpiCC --download-f-blas-lapack --prefix=/opt/petsc3.3-1 --with-debugging
> [7]PETSC ERROR: ------------------------------------------------------------------------
> [7]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
> --------------------------------------------------------------------------
> mpiexec has exited due to process rank 0 with PID 2670 on
> node grsacc17 exiting improperly. There are two reasons this could occur:
>
> 1. this process did not call "init" before exiting, but others in
> the job did. This can cause a job to hang indefinitely while it waits
> for all processes to call "init". By rule, if one process calls "init",
> then ALL processes must call "init" prior to termination.
>
> 2. this process called "init", but exited without calling "finalize".
> By rule, all processes that call "init" MUST call "finalize" prior to
> exiting or it will be considered an "abnormal termination"
>
> This may have caused other processes in the application to be
> terminated by signals sent by mpiexec (as reported here).
More information about the petsc-users
mailing list