3D POISEUILLE BASEFLOW COMPUTED ON GRID WITH NX =101 NY =101 Time for build and assembly of baseflow operator = 0.0720 seconds. Time for Matrix summation = 0.0480 seconds. [5]PETSC ERROR: [21]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: ------------------------------------------------------------------------ Caught signal number 8 FPE: Floating Point Exception,probably divide by zero [21]PETSC ERROR: [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point Exception,probably divide by zero Try option -start_in_debugger or -on_error_attach_debugger [21]PETSC ERROR: [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [21]PETSC ERROR: [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [21]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [5]PETSC ERROR: [21]PETSC ERROR: likely location of problem given in stack below likely location of problem given in stack below [5]PETSC ERROR: [21]PETSC ERROR: --------------------- Stack Frames ------------------------------------ --------------------- Stack Frames ------------------------------------ [5]PETSC ERROR: [21]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, Note: The EXACT line numbers in the stack are not available, [5]PETSC ERROR: [21]PETSC ERROR: INSTEAD the line number of the start of the function INSTEAD the line number of the start of the function [5]PETSC ERROR: [21]PETSC ERROR: is given. is given. [5]PETSC ERROR: [21]PETSC ERROR: [5] SuperLU_DIST:pzgssvx line 411 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [21] SuperLU_DIST:pzgssvx line 411 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [5]PETSC ERROR: [21]PETSC ERROR: [5] MatLUFactorNumeric_SuperLU_DIST line 282 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [21] MatLUFactorNumeric_SuperLU_DIST line 282 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c [5]PETSC ERROR: [21]PETSC ERROR: [5] MatLUFactorNumeric line 2937 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/interface/matrix.c [21] MatLUFactorNumeric line 2937 /lustre/home1/u/anthony/petsc-3.6.0/src/mat/interface/matrix.c [5]PETSC ERROR: [21]PETSC ERROR: [5] PCSetUp_LU line 99 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c [21] PCSetUp_LU line 99 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/pc/impls/factor/lu/lu.c [5]PETSC ERROR: [21]PETSC ERROR: [5] PCSetUp line 944 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/pc/interface/precon.c [21] PCSetUp line 944 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/pc/interface/precon.c [5]PETSC ERROR: [21]PETSC ERROR: [5] KSPSetUp line 247 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [21] KSPSetUp line 247 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [5]PETSC ERROR: [21]PETSC ERROR: [5] KSPSolve line 510 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [21] KSPSolve line 510 /lustre/home1/u/anthony/petsc-3.6.0/src/ksp/ksp/interface/itfunc.c [5]PETSC ERROR: [21]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: [21]PETSC ERROR: Signal received Signal received [5]PETSC ERROR: [21]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: [21]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 Petsc Release Version 3.6.0, Jun, 09, 2015 [5]PETSC ERROR: [21]PETSC ERROR: BiGlobal_Spectral_Petsc.x on a linux-gnu-debug-mumps-superludist-complex named nid04601 by anthony Wed Jul 8 12:17:34 2015 BiGlobal_Spectral_Petsc.x on a linux-gnu-debug-mumps-superludist-complex named nid04601 by anthony Wed Jul 8 12:17:34 2015 [5]PETSC ERROR: [21]PETSC ERROR: Configure options PETSC_ARCH=linux-gnu-debug-mumps-superludist-complex CC=cc CXX=CC FC=ftn --useThreads=1 --with-x=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-scalar-type=complex --download-fblaslapack --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist Configure options PETSC_ARCH=linux-gnu-debug-mumps-superludist-complex CC=cc CXX=CC FC=ftn --useThreads=1 --with-x=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-scalar-type=complex --download-fblaslapack --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist [5]PETSC ERROR: [21]PETSC ERROR: #1 User provided function() line 0 in unknown file #1 User provided function() line 0 in unknown file Rank 5 [Wed Jul 8 12:17:44 2015] [c7-0c0s3n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 Program received signal SIGABRT: Process abort signal. Rank 21 [Wed Jul 8 12:17:44 2015] [c7-0c0s3n1] application called MPI_Abort(MPI_COMM_WORLD, 59) - process 21 Backtrace for this error: Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x20600AD in _gfortrani_backtrace at backtrace.c:258 #0 0x20600AD in _gfortrani_backtrace at backtrace.c:258 #1 0x204BEE0 in _gfortrani_backtrace_handler at compile_options.c:129 #1 0x204BEE0 in _gfortrani_backtrace_handler at compile_options.c:129 #2 0x221693F in raise #2 0x221693F in raise #3 0x22168FB in raise at pt-raise.c:41 #3 0x22168FB in raise at pt-raise.c:41 #4 0x2227320 in abort at abort.c:92 #4 0x2227320 in abort at abort.c:92 #5 0x20F3361 in MPID_Abort #5 0x20F3361 in MPID_Abort #6 0x20C83B2 in PMPI_Abort #6 0x20C83B2 in PMPI_Abort #7 0x4D35AA in PetscSignalHandlerDefault at signal.c:161 #7 0x4D35AA in PetscSignalHandlerDefault at signal.c:161 #8 0x4D2DFD in PetscSignalHandler_Private at signal.c:49 #8 0x4D2DFD in PetscSignalHandler_Private at signal.c:49 #9 0x221693F in raise #9 0x221693F in raise #10 0x1D54440 in libmetis__FM_2WayCutRefine at fm.c:60 #11 0x1D541FD in libmetis__FM_2WayRefine at fm.c:20 #10 0x1D54440 in libmetis__FM_2WayCutRefine at fm.c:60 #11 0x1D541FD in libmetis__FM_2WayRefine at fm.c:20 #12 0x1D58CDD in libmetis__RandomBisection at initpart.c:165 #13 0x1D589C0 in libmetis__InitSeparator at initpart.c:84 #12 0x1D58CDD in libmetis__RandomBisection at initpart.c:165 #13 0x1D589C0 in libmetis__InitSeparator at initpart.c:84 #14 0x1D3DE13 in libmetis__MlevelNodeBisectionL1 at ometis.c:409 #15 0x1D3DBB5 in libmetis__MlevelNodeBisectionL2 at ometis.c:353 #14 0x1D3DE13 in libmetis__MlevelNodeBisectionL1 at ometis.c:409 #16 0x1D3DA54 in libmetis__MlevelNodeBisectionMultiple at ometis.c:307 #15 0x1D3DBB5 in libmetis__MlevelNodeBisectionL2 at ometis.c:353 #16 0x1D3DA54 in libmetis__MlevelNodeBisectionMultiple at ometis.c:307 #17 0x1D41207 in METIS_ComputeVertexSeparator at parmetis.c:182 #17 0x1D41207 in METIS_ComputeVertexSeparator at parmetis.c:182 #18 0x1D10475 in libparmetis__InitMultisection at initmsection.c:82 #18 0x1D10475 in libparmetis__InitMultisection at initmsection.c:82 #19 0x1D00FA0 in libparmetis__Order_Partition at ometis.c:360 #20 0x1D00B15 in libparmetis__Order_Partition_Multiple at ometis.c:302 #19 0x1D00FA0 in libparmetis__Order_Partition at ometis.c:360 #21 0x1D00663 in libparmetis__MultilevelOrder at ometis.c:221 #20 0x1D00B15 in libparmetis__Order_Partition_Multiple at ometis.c:302 #22 0x1D003E2 in ParMETIS_V32_NodeND at ometis.c:151 #23 0x1CFFE73 in #21 ParMETIS_V3_NodeND0x at 1D00663ometis.c:34 in libparmetis__MultilevelOrder at ometis.c:221 #22 0x1D003E2 in ParMETIS_V32_NodeND at ometis.c:151 #23 0x1CFFE73 in ParMETIS_V3_NodeND at ometis.c:34 #24 0x1C47591 in get_perm_c_parmetis at get_perm_c_parmetis.c:241 #24 0x1C47591 in get_perm_c_parmetis at get_perm_c_parmetis.c:241 #25 0x1C0310A in pzgssvx at pzgssvx.c:928 #25 0x1C0310A in pzgssvx at pzgssvx.c:928 #26 0x105FA5E in MatLUFactorNumeric_SuperLU_DIST at superlu_dist.c:411 #26 0x105FA5E in MatLUFactorNumeric_SuperLU_DIST at superlu_dist.c:411 #27 0x10ACBB1 in MatLUFactorNumeric at matrix.c:2948 #27 0x10ACBB1 in MatLUFactorNumeric at matrix.c:2948 #28 0x133E54A in PCSetUp_LU at lu.c:152 #28 0x133E54A in PCSetUp_LU at lu.c:152 #29 0x1242D12 in PCSetUp at precon.c:982 #29 0x1242D12 in PCSetUp at precon.c:982 #30 0x1413A69 in KSPSetUp at itfunc.c:332 #31 0x1415685 in KSPSolve at itfunc.c:546 #30 0x1413A69 in KSPSetUp at itfunc.c:332 #31 0x1415685 in KSPSolve at itfunc.c:546 #32 0x77A343 in kspsolve_ at itfuncf.c:247 #32 0x77A343 in kspsolve_ at itfuncf.c:247 #33 0x4114C1 in __baseflow_MOD_poiseuille_3d_petsc at module_baseflows.F90:314 #33 0x4114C1 in __baseflow_MOD_poiseuille_3d_petsc at module_baseflows.F90:314 #34 0x417541 in biglobal_spectral at BiGlobal_Spectral_Petsc.F90:246 #34 0x417541 in biglobal_spectral at BiGlobal_Spectral_Petsc.F90:246 _pmiu_daemon(SIGCHLD): [NID 04601] [c7-0c0s3n1] [Wed Jul 8 12:17:44 2015] PE RANK 21 exit signal Aborted [NID 04601] 2015-07-08 12:17:44 Apid 31033945: initiated application termination Application 31033945 exit codes: 134 Application 31033945 resources: utime ~10s, stime ~2s, Rss ~106848, inblocks ~71051, outblocks ~111077