[petsc-users] A problem with sizeof grid

Sat Aug 24 02:54:59 CDT 2013

Hi,

I ran my 3d structured CFD code for a cavity flow with different grid size from 20x20x20 to 80x80x80. Whether fast or not, all they converged and gave me the expected result.

but when I tried with 120x120x120 grid, errors appeared as follows. Since I believe my physical memory is large enough for these test. So I just wonder what really happened with the GMRES solver + hypre PC. I hope you may leave me a hint. Thanks.

Zhenyu
mpirun -np 8 ./nsproj  ../input/sample_120.zzy.dat -px 2 -py 2 -pz 2 -ksp_type gmres -pc_type hypre -ksp_rtol 1.e-2

....GRID POINT DIMENSIONS W/ GHOST CELLS : 122 X 122 X 122
px= 2, py= 2, pz= 2
....BEGINNING SIMULATION
time: 0,  SolvePressure iter= 0, residnorm= 0.000000e+00
time: 20,  SolvePressure iter= 2, residnorm= 1.329163e+21
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in external library!
[0]PETSC ERROR: Error in HYPRE solver, error code 1!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[0]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[0]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[1]PETSC ERROR: --------------------- Error Message ------------------------------------
[1]PETSC ERROR: Error in external library!
[1]PETSC ERROR: Error in HYPRE solver, error code 1!
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[1]PETSC ERROR: See docs/changes/index.html for recent updates.
[1]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[1]PETSC ERROR: See docs/index.html for manual pages.
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[1]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[1]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[1]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[1]PETSC ERROR: ------------------------------------------------------------------------
[1]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message ------------------------------------
[2]PETSC ERROR: Error in external library!
[2]PETSC ERROR: Error in HYPRE solver, error code 1!
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[2]PETSC ERROR: See docs/changes/index.html for recent updates.
[2]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[2]PETSC ERROR: See docs/index.html for manual pages.
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[2]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[2]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[2]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[2]PETSC ERROR: ------------------------------------------------------------------------
[2]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[2]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[2]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[2]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[3]PETSC ERROR: --------------------- Error Message ------------------------------------
[3]PETSC ERROR: Error in external library!
[3]PETSC ERROR: Error in HYPRE solver, error code 1!
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[3]PETSC ERROR: See docs/changes/index.html for recent updates.
[3]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[3]PETSC ERROR: See docs/index.html for manual pages.
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[3]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[3]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[3]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[3]PETSC ERROR: ------------------------------------------------------------------------
[3]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[3]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message ------------------------------------
[4]PETSC ERROR: Error in external library!
[4]PETSC ERROR: Error in HYPRE solver, error code 1!
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[4]PETSC ERROR: See docs/changes/index.html for recent updates.
[4]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[4]PETSC ERROR: See docs/index.html for manual pages.
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[4]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[4]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[4]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[4]PETSC ERROR: ------------------------------------------------------------------------
[4]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[4]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[4]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[4]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[4]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[4]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[4]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[4]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
[5]PETSC ERROR: --------------------- Error Message ------------------------------------
[5]PETSC ERROR: Error in external library!
[5]PETSC ERROR: Error in HYPRE solver, error code 1!
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[5]PETSC ERROR: See docs/changes/index.html for recent updates.
[5]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[5]PETSC ERROR: See docs/index.html for manual pages.
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[5]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[5]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[5]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[5]PETSC ERROR: ------------------------------------------------------------------------
[5]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[5]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[5]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[5]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[5]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[5]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[5]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[5]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
[6]PETSC ERROR: --------------------- Error Message ------------------------------------
[6]PETSC ERROR: Error in external library!
[6]PETSC ERROR: Error in HYPRE solver, error code 1!
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[6]PETSC ERROR: See docs/changes/index.html for recent updates.
[6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[6]PETSC ERROR: See docs/index.html for manual pages.
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[6]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[6]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[6]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[6]PETSC ERROR: ------------------------------------------------------------------------
[6]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[6]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[6]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[6]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[6]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[7]PETSC ERROR: --------------------- Error Message ------------------------------------
[7]PETSC ERROR: Error in external library!
[7]PETSC ERROR: Error in HYPRE solver, error code 1!
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: Petsc Release Version 3.4.2, Jul, 02, 2013
[7]PETSC ERROR: See docs/changes/index.html for recent updates.
[7]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[7]PETSC ERROR: See docs/index.html for manual pages.
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: ./nsproj on a arch-linux2-c-opt named amax by zhenyu Sat Aug 24 16:05:05 2013
[7]PETSC ERROR: Libraries linked from /home/zhenyu/petsc-3.4.2/arch-linux2-c-opt/lib
[7]PETSC ERROR: Configure run at Sat Aug 24 12:43:12 2013
[7]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[7]PETSC ERROR: ------------------------------------------------------------------------
[7]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[7]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[7]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[7]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[7]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[7]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[7]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[7]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
[0]PETSC ERROR: Configure options --with-debugging=0 --with-shared-libraries=1 --with-dynamic-loading=1 --with-x=1 --with-blas-lapack-dir=/usr/lib/lapack --with-valgrind=1 --with-cc=gcc --with-fc=gfortran --with-clanguage=C++ --with-c++-support=1 --with-sieve=1 --with-opt-sieve=1 --download-generator --download-triangle --download-ctetgen --with-ctetgen --download-chaco --download-boost=1 --download-hypre=1 --download-metis --download-parmetis --download-hdf5 --download-openmpi --download-f-blas-lapack --download-superlu_dist
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PCApply_HYPRE() line 170 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/impls/hypre/hypre.c
[0]PETSC ERROR: PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[0]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[0]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
PCApply() line 442 in /home/zhenyu/petsc-3.4.2/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[1]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[1]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[1]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[1]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
[2]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[2]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[2]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[2]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
KSP_PCApply() line 227 in /home/zhenyu/petsc-3.4.2/include/petsc-private/kspimpl.h
[3]PETSC ERROR: KSPInitialResidual() line 64 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itres.c
[3]PETSC ERROR: KSPSolve_GMRES() line 239 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/impls/gmres/gmres.c
[3]PETSC ERROR: KSPSolve() line 441 in /home/zhenyu/petsc-3.4.2/src/ksp/ksp/interface/itfunc.c
[3]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[3]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
[6]PETSC ERROR: SolvePressure() line 898 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nssolver_petsc.cc
[6]PETSC ERROR: start_sim() line 190 in "unknowndirectory/"/home/zhenyu/work/nsproj_0.3_zzy/src/nsproj_petsc.cc
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130824/54d1f483/attachment-0001.html>