[petsc-users] Error - Out of memory. This could be due to allocating too large an object or bleeding by not properly ...
TAY wee-beng
zonexo at gmail.com
Tue Feb 23 19:50:52 CST 2016
Hi,
I got this error (also attached, full) when running my code. It happens
after a few thousand time steps.
The strange thing is that for 2 different clusters, it stops at 2
different time steps.
I wonder if it's related to DM since this happens after I added DM into
my code.
In this case, how can I find out the error? I'm thinking valgrind may
take very long and gives too many false errors.
--
Thank you
Yours sincerely,
TAY wee-beng
-------------- next part --------------
****************************************************************************
* hwloc has encountered what looks like an error from the operating system.
*
* L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion!
* Error occurred in topology.c line 942
*
* The following FAQ entry in a recent hwloc documentation may help:
* What should I do when hwloc reports "operating system" warnings?
* Otherwise please report this error message to the hwloc user's mailing list,
* along with the output+tarball generated by the hwloc-gather-topology script.
****************************************************************************
****************************************************************************
* hwloc has encountered what looks like an error from the operating system.
*
* L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion!
* Error occurred in topology.c line 942
*
* The following FAQ entry in a recent hwloc documentation may help:
* What should I do when hwloc reports "operating system" warnings?
* Otherwise please report this error message to the hwloc user's mailing list,
* along with the output+tarball generated by the hwloc-gather-topology script.
****************************************************************************
[35]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[35]PETSC ERROR: Out of memory. This could be due to allocating
[35]PETSC ERROR: too large an object or bleeding by not properly
[35]PETSC ERROR: destroying unneeded objects.
[46]PETSC ERROR: [19]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[19]PETSC ERROR: Out of memory. This could be due to allocating
[19]PETSC ERROR: too large an object or bleeding by not properly
[19]PETSC ERROR: destroying unneeded objects.
[19]PETSC ERROR: Memory allocated 0 Memory used by process 3551756288
[19]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[19]PETSC ERROR: Memory requested 34905645
[19]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[19]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[32]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[32]PETSC ERROR: Out of memory. This could be due to allocating
[32]PETSC ERROR: too large an object or bleeding by not properly
[32]PETSC ERROR: destroying unneeded objects.
[35]PETSC ERROR: Memory allocated 0 Memory used by process 3522326528
[35]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[35]PETSC ERROR: Memory requested 34905645
[35]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[35]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[35]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[35]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[35]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[35]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[35]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[35]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[35]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[35]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[35]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[35]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[35]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[35]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[35]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[35]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[35]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
--------------------- Error Message --------------------------------------------------------------
[46]PETSC ERROR: Out of memory. This could be due to allocating
[46]PETSC ERROR: too large an object or bleeding by not properly
[46]PETSC ERROR: destroying unneeded objects.
[46]PETSC ERROR: Memory allocated 0 Memory used by process 3510972416
[46]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[46]PETSC ERROR: Memory requested 34905645
[46]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[46]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[46]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[46]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[46]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[46]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[46]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[46]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[46]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[19]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[19]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[19]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[34]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[34]PETSC ERROR: Out of memory. This could be due to allocating
[34]PETSC ERROR: too large an object or bleeding by not properly
[34]PETSC ERROR: destroying unneeded objects.
[19]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[19]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[19]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[19]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[19]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[19]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[19]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[19]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[19]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[19]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[19]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: Memory allocated 0 Memory used by process 3514494976
[32]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[32]PETSC ERROR: Memory requested 34905645
[32]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[32]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[32]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[32]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[32]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[32]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[32]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[32]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[32]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[32]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[32]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[32]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[32]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[32]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[32]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[34]PETSC ERROR: Memory allocated 0 Memory used by process 3541282816
[34]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[34]PETSC ERROR: Memory requested 34905645
[34]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[34]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[34]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[34]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[34]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[34]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[34]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[34]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[34]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[34]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[34]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[34]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[34]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[34]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[34]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[34]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[34]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[46]PETSC ERROR: [33]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[33]PETSC ERROR: Out of memory. This could be due to allocating
[33]PETSC ERROR: too large an object or bleeding by not properly
[33]PETSC ERROR: destroying unneeded objects.
#6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[33]PETSC ERROR: Memory allocated 0 Memory used by process 3514351616
[33]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[33]PETSC ERROR: Memory requested 34905645
[33]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[33]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[33]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[33]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[33]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[33]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1219 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[33]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[33]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[33]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[33]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[33]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[33]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[33]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[33]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[33]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[33]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[33]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[46]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[46]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[46]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[46]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[46]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[46]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[46]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[16]PETSC ERROR: Out of memory. This could be due to allocating
[12]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[12]PETSC ERROR: Out of memory. This could be due to allocating
[12]PETSC ERROR: too large an object or bleeding by not properly
[12]PETSC ERROR: destroying unneeded objects.
[12]PETSC ERROR: Memory allocated 0 Memory used by process 3533832192
[12]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[12]PETSC ERROR: Memory requested 4363204
[12]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[12]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[12]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[12]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[12]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[12]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[12]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[12]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[12]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[12]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[12]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[12]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[12]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[12]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[12]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[12]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[12]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: too large an object or bleeding by not properly
[16]PETSC ERROR: destroying unneeded objects.
[16]PETSC ERROR: Memory allocated 0 Memory used by process 3527131136
[16]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
[16]PETSC ERROR: Memory requested 4363204
[16]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[16]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015
[16]PETSC ERROR: ./a.out on a petsc-3.6.3_static_rel named n12-03 by wtay Sun Feb 21 08:24:37 2016
[16]PETSC ERROR: Configure options --with-mpi-dir=/opt/ud/openmpi-1.8.8/ --with-blas-lapack-dir=/opt/ud/intel_xe_2013sp1/mkl/lib/intel64/ --with-debugging=0 --download-hypre=1 --prefix=/home/wtay/Lib/petsc-3.6.3_static_rel --known-mpi-shared=0 --with-shared-libraries=0 --with-fortran-interfaces=1
[16]PETSC ERROR: #1 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[16]PETSC ERROR: #2 MatLUFactorNumeric_SeqAIJ_Inode() line 1228 in /home/wtay/Codes/petsc-3.6.3/src/mat/impls/aij/seq/inode.c
[16]PETSC ERROR: #3 MatLUFactorNumeric() line 2946 in /home/wtay/Codes/petsc-3.6.3/src/mat/interface/matrix.c
[16]PETSC ERROR: #4 PCSetUp_ILU() line 233 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/factor/ilu/ilu.c
[16]PETSC ERROR: #5 PCSetUp() line 983 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[16]PETSC ERROR: #6 KSPSetUp() line 332 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: #7 KSPSolve() line 546 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[16]PETSC ERROR: #8 PCApply_BJacobi_Singleblock() line 670 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/impls/bjacobi/bjacobi.c
[16]PETSC ERROR: #9 PCApply() line 483 in /home/wtay/Codes/petsc-3.6.3/src/ksp/pc/interface/precon.c
[16]PETSC ERROR: #10 KSP_PCApply() line 242 in /home/wtay/Codes/petsc-3.6.3/include/petsc/private/kspimpl.h
[16]PETSC ERROR: #11 KSPInitialResidual() line 63 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itres.c
[16]PETSC ERROR: #12 KSPSolve_BCGS() line 50 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c
[16]PETSC ERROR: #13 KSPSolve() line 604 in /home/wtay/Codes/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
mpiexec: abort is already in progress...hit ctrl-c again to forcibly terminate
-------------- next part --------------
0.000000000000000E+000 0.710000000000000 0.000000000000000E+000
90.0000000000000 0.000000000000000E+000 -0.350000000000000
1.00000000000000 0.200000000000000 0 -400000
AB,AA,BB,CC -3.13300005381461 3.47950007027248
3.69050008029444 3.79850008542417
size_x,size_y,size_z 202 300 577
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 0 1 50
1 37 1 373700
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 1 51 100
1 37 373701 747400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 2 101 150
1 37 747401 1121100
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 3 151 200
1 37 1121101 1494800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 4 201 250
1 37 1494801 1868500
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 5 251 300
1 37 1868501 2242200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 6 1 50
38 73 2242201 2605800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 7 51 100
38 73 2605801 2969400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 8 101 150
38 73 2969401 3333000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 9 151 200
38 73 3333001 3696600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 10 201 250
38 73 3696601 4060200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 11 251 300
38 73 4060201 4423800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 12 1 50
74 109 4423801 4787400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 13 51 100
74 109 4787401 5151000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 14 101 150
74 109 5151001 5514600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 15 151 200
74 109 5514601 5878200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 16 201 250
74 109 5878201 6241800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 17 251 300
74 109 6241801 6605400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 18 1 50
110 145 6605401 6969000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 19 51 100
110 145 6969001 7332600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 20 101 150
110 145 7332601 7696200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 21 151 200
110 145 7696201 8059800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 22 201 250
110 145 8059801 8423400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 23 251 300
110 145 8423401 8787000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 24 1 50
146 181 8787001 9150600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 25 51 100
146 181 9150601 9514200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 26 101 150
146 181 9514201 9877800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 27 151 200
146 181 9877801 10241400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 28 201 250
146 181 10241401 10605000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 29 251 300
146 181 10605001 10968600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 30 1 50
182 217 10968601 11332200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 31 51 100
182 217 11332201 11695800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 32 101 150
182 217 11695801 12059400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 33 151 200
182 217 12059401 12423000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 34 201 250
182 217 12423001 12786600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 35 251 300
182 217 12786601 13150200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 36 1 50
218 253 13150201 13513800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 37 51 100
218 253 13513801 13877400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 38 101 150
218 253 13877401 14241000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 39 151 200
218 253 14241001 14604600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 40 201 250
218 253 14604601 14968200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 41 251 300
218 253 14968201 15331800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 42 1 50
254 289 15331801 15695400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 43 51 100
254 289 15695401 16059000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 44 101 150
254 289 16059001 16422600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 45 151 200
254 289 16422601 16786200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 46 201 250
254 289 16786201 17149800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 47 251 300
254 289 17149801 17513400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 48 1 50
290 325 17513401 17877000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 49 51 100
290 325 17877001 18240600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 50 101 150
290 325 18240601 18604200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 51 151 200
290 325 18604201 18967800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 52 201 250
290 325 18967801 19331400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 53 251 300
290 325 19331401 19695000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 54 1 50
326 361 19695001 20058600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 55 51 100
326 361 20058601 20422200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 56 101 150
326 361 20422201 20785800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 57 151 200
326 361 20785801 21149400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 58 201 250
326 361 21149401 21513000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 59 251 300
326 361 21513001 21876600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 60 1 50
362 397 21876601 22240200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 61 51 100
362 397 22240201 22603800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 62 101 150
362 397 22603801 22967400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 63 151 200
362 397 22967401 23331000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 64 201 250
362 397 23331001 23694600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 65 251 300
362 397 23694601 24058200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 66 1 50
398 433 24058201 24421800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 67 51 100
398 433 24421801 24785400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 68 101 150
398 433 24785401 25149000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 69 151 200
398 433 25149001 25512600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 70 201 250
398 433 25512601 25876200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 71 251 300
398 433 25876201 26239800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 72 1 50
434 469 26239801 26603400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 73 51 100
434 469 26603401 26967000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 74 101 150
434 469 26967001 27330600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 75 151 200
434 469 27330601 27694200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 76 201 250
434 469 27694201 28057800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 77 251 300
434 469 28057801 28421400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 78 1 50
470 505 28421401 28785000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 79 51 100
470 505 28785001 29148600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 80 101 150
470 505 29148601 29512200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 81 151 200
470 505 29512201 29875800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 82 201 250
470 505 29875801 30239400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 83 251 300
470 505 30239401 30603000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 84 1 50
506 541 30603001 30966600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 85 51 100
506 541 30966601 31330200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 86 101 150
506 541 31330201 31693800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 87 151 200
506 541 31693801 32057400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 88 201 250
506 541 32057401 32421000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 89 251 300
506 541 32421001 32784600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 90 1 50
542 577 32784601 33148200
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 91 51 100
542 577 33148201 33511800
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 92 101 150
542 577 33511801 33875400
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 93 151 200
542 577 33875401 34239000
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 94 201 250
542 577 34239001 34602600
myid,jsta,jend,ksta,kend,ijk_sta,ijk_end 95 251 300
542 577 34602601 34966200
body_cg_ini 9.999999999255602E-007 1.710991388089775E-003
10.2254852004451
672 surfaces with correct vertex ordering
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.297282883542864 5.747832995728071E-002 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 1 1
minimum ngh_surfaces and ngh_vertics are 1 1
body_cg_ini -9.199712417665717E-004 0.891121393810939
6.75773429502958
1776 surfaces with correct vertex ordering
Warning - length difference between element and cell
max_element_length,min_element_length,min_delta
0.294864814733624 1.648655999999882E-002 1.800000000000000E-002
maximum ngh_surfaces and ngh_vertics are 1 2
minimum ngh_surfaces and ngh_vertics are 1 1
min IIB_cell_no 0
max IIB_cell_no 1237
final initial IIB_cell_no 61850
min I_cell_no 0
max I_cell_no 4428
final initial I_cell_no 221400
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
61850 221400 61850 221400
min IIB_cell_no 0
max IIB_cell_no 1350
final initial IIB_cell_no 67500
min I_cell_no 0
max I_cell_no 22199
final initial I_cell_no 1109950
size(IIB_cell_u),size(I_cell_u),size(IIB_equal_cell_u),size(I_equal_cell_u)
67500 1109950 67500 1109950
IIB_I_cell_no_uvw_total1 18956 18602 18748 57477
49290 49710
IIB_I_cell_no_uvw_total2 37851 37876 38095 500842
504527 506410
1500 0.00021825 9.98988642 10.43276011 8.24268002 0.64333010E+06 0.41880252E+04 0.34216012E+08
1800 0.00023513 10.68688767 9.49082300 8.14022395 0.47192748E+06 0.37796673E+04 0.34397558E+08
2100 0.00023612 11.14050288 9.53122678 8.10449896 0.24054404E+06 0.22516421E+04 0.34588837E+08
--------------------------------------------------------------------------
ORTE has lost communication with its daemon located on node:
hostname: n12-09
This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------
More information about the petsc-users
mailing list