[petsc-users] Debug AOCreateBasic

[Rongliang Chen]

Hi there,

My code died in the AOCreateBasic and the error messages are followed. 
Do you have any suggestions to debug this?

Notes:
1. In this case, it has about 30,000,000 unstructured mesh and use 96 
processors (I also tried 1024 processors and it has the same problem).
2. My code works well for a smaller case (about 25,000,000 unstructured 
meshes) .
3. I also check the memory usage of this case and it is very small 
because the solution stage does not start yet.

Best,
Rongliang

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the 
batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] AOCreate_Basic line 203 
src/vec/is/ao/impls/basic/aobasic.c
[0]PETSC ERROR: [0] AOSetType line 35 src/vec/is/ao/interface/aoreg.c
[0]PETSC ERROR: [0] AOCreateBasicIS line 380 
src/vec/is/ao/impls/basic/aobasic.c
[0]PETSC ERROR: [0] AOCreateBasic line 335 
src/vec/is/ao/impls/basic/aobasic.c
[0]PETSC ERROR: [0] DataPartitionVertices_Block line 1634 
/projects/ronglian/soft/3Dfluid_new/3DWindturbine/WindturbineFor3.4/codes/readbinary3d.c
[0]PETSC ERROR: [0] ReadBinary line 184 
/projects/ronglian/soft/3Dfluid_new/3DWindturbine/WindturbineFor3.4/codes/readbinary3d.c
[0]PETSC ERROR: [0] LoadGrid line 720 
/projects/ronglian/soft/3Dfluid_new/3DWindturbine/WindturbineFor3.4/codes/loadgrid3d.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision: 
ee17fca9fd6ac48e6579ef235144daafbb22b801  GIT Date: 2013-10-23 14:21:20 
-0500
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: ./fsi3d on a Janus-debug-64bit named node0880 by 
ronglian Thu Oct 31 21:58:20 2013
[0]PETSC ERROR: Libraries linked from 
/projects/ronglian/soft/petsc-dev-latest/Janus-debug-64bit/lib
[0]PETSC ERROR: Configure run at Thu Oct 24 21:24:31 2013
[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768 
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8 
--known-memcmp-ok=1 --known-sizeof-char=1 --known-sizeof-void-p=8 
--known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 
--known-sizeof-long-long=8 --known-sizeof-float=4 
--known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 
--known-sizeof-MPI_Comm=8 --known-sizeof-MPI_Fint=4 
--known-mpi-long-double=1 --known-mpi-c-double-complex=0 
--download-blacs=1 --download-f-blas-lapack=1 --download-metis=1 
--download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 
--known-mpi-shared-libraries=0 --with-64-bit-indices --with-batch=1 
--with-mpi-shared=1 --download-exodusii=1 --download-hdf5=1 
--download-netcdf=1 --known-64-bit-blas-indices --with-debugging=1 
COPTFLAGS="-O0 -g"
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in  unknown file
[1]PETSC ERROR: 
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the 
batch system) has told this process to end
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[1]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below