[petsc-dev] GMRES with one processor crashes

Alexander Grayver agrayver at gfz-potsdam.de
Mon Jan 9 05:23:03 CST 2012 

Hi dev team,

I run petsc-dev like that:
mpirun -n 2 /home/solveTest -ksp_view -ksp_monitor -ksp_converged_reason 
-ksp_monitor_true_residual -log_summary -mat_type mpiaij -ksp_rtol 1.0e-12

And it works perfect:
   0 KSP preconditioned resid norm 4.771657217422e-19 true resid norm 
5.118310922721e-16 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 1.279447738882e-19 true resid norm 
1.285092439316e-16 ||r(i)||/||b|| 2.510774469780e-01
   2 KSP preconditioned resid norm 5.321531301146e-20 true resid norm 
5.433643516486e-17 ||r(i)||/||b|| 1.061608721808e-01
   3 KSP preconditioned resid norm 2.271494567767e-20 true resid norm 
2.372382040456e-17 ||r(i)||/||b|| 4.635087778517e-02
   4 KSP preconditioned resid norm 1.086064397433e-20 true resid norm 
1.155433840255e-17 ||r(i)||/||b|| 2.257451447754e-02
   5 KSP preconditioned resid norm 5.222418228186e-21 true resid norm 
5.609167739223e-18 ||r(i)||/||b|| 1.095902109878e-02
   6 KSP preconditioned resid norm 2.635275019546e-21 true resid norm 
2.908236562671e-18 ||r(i)||/||b|| 5.682024024294e-03
   7 KSP preconditioned resid norm 1.356113681173e-21 true resid norm 
1.522329163123e-18 ||r(i)||/||b|| 2.974280355586e-03
   8 KSP preconditioned resid norm 7.131139215070e-22 true resid norm 
8.193371730543e-19 ||r(i)||/||b|| 1.600796015375e-03
   9 KSP preconditioned resid norm 3.803513444633e-22 true resid norm 
4.413453890272e-19 ||r(i)||/||b|| 8.622871796786e-04
  10 KSP preconditioned resid norm 2.054204434835e-22 true resid norm 
2.419916966599e-19 ||r(i)||/||b|| 4.727960069516e-04
  11 KSP preconditioned resid norm 1.117647186186e-22 true resid norm 
1.328122348588e-19 ||r(i)||/||b|| 2.594844996017e-04
  12 KSP preconditioned resid norm 6.119052195598e-23 true resid norm 
7.312625584441e-20 ||r(i)||/||b|| 1.428718515708e-04
  13 KSP preconditioned resid norm 3.364628967843e-23 true resid norm 
4.038708166230e-20 ||r(i)||/||b|| 7.890705014228e-05
  14 KSP preconditioned resid norm 1.862114435304e-23 true resid norm 
2.238513973729e-20 ||r(i)||/||b|| 4.373540426768e-05
  15 KSP preconditioned resid norm 1.048192086360e-23 true resid norm 
1.255409141760e-20 ||r(i)||/||b|| 2.452780147035e-05
  16 KSP preconditioned resid norm 6.057516927187e-24 true resid norm 
7.215197109086e-21 ||r(i)||/||b|| 1.409683236916e-05
  17 KSP preconditioned resid norm 3.583631444956e-24 true resid norm 
4.269063224215e-21 ||r(i)||/||b|| 8.340765710938e-06
  18 KSP preconditioned resid norm 2.084932321415e-24 true resid norm 
2.503288447972e-21 ||r(i)||/||b|| 4.890848730700e-06
  19 KSP preconditioned resid norm 1.197296409471e-24 true resid norm 
1.444680159565e-21 ||r(i)||/||b|| 2.822572097275e-06
  20 KSP preconditioned resid norm 6.915346923639e-25 true resid norm 
8.294994358759e-22 ||r(i)||/||b|| 1.620650734979e-06
  21 KSP preconditioned resid norm 4.055788682943e-25 true resid norm 
4.864602674903e-22 ||r(i)||/||b|| 9.504312552229e-07
  22 KSP preconditioned resid norm 2.379775725935e-25 true resid norm 
2.861428158451e-22 ||r(i)||/||b|| 5.590571189703e-07
  23 KSP preconditioned resid norm 1.382986241145e-25 true resid norm 
1.667791605604e-22 ||r(i)||/||b|| 3.258480445571e-07
  24 KSP preconditioned resid norm 8.045844789016e-26 true resid norm 
9.695300855633e-23 ||r(i)||/||b|| 1.894238353632e-07
  25 KSP preconditioned resid norm 4.714035145740e-26 true resid norm 
5.664525004251e-23 ||r(i)||/||b|| 1.106717643726e-07
  26 KSP preconditioned resid norm 2.766941968338e-26 true resid norm 
3.335733065645e-23 ||r(i)||/||b|| 6.517253672177e-08
  27 KSP preconditioned resid norm 1.612427411739e-26 true resid norm 
1.945526739209e-23 ||r(i)||/||b|| 3.801110891041e-08
  28 KSP preconditioned resid norm 9.395679680467e-27 true resid norm 
1.133388453345e-23 ||r(i)||/||b|| 2.214379842212e-08
  29 KSP preconditioned resid norm 5.486123335914e-27 true resid norm 
6.613765802620e-24 ||r(i)||/||b|| 1.292177419949e-08
  30 KSP preconditioned resid norm 3.215178475481e-27 true resid norm 
3.871477901985e-24 ||r(i)||/||b|| 7.563975617031e-09
  31 KSP preconditioned resid norm 2.250287962378e-27 true resid norm 
3.004475624127e-24 ||r(i)||/||b|| 5.870052971557e-09
  32 KSP preconditioned resid norm 1.376915678600e-27 true resid norm 
1.807238225196e-24 ||r(i)||/||b|| 3.530927004011e-09
  33 KSP preconditioned resid norm 8.352384279993e-28 true resid norm 
1.049868218945e-24 ||r(i)||/||b|| 2.051200551893e-09
  34 KSP preconditioned resid norm 4.874692135062e-28 true resid norm 
6.092834522927e-25 ||r(i)||/||b|| 1.190399453046e-09
  35 KSP preconditioned resid norm 2.843218847295e-28 true resid norm 
3.432278384183e-25 ||r(i)||/||b|| 6.705880975198e-10
  36 KSP preconditioned resid norm 1.679613904081e-28 true resid norm 
2.032631030186e-25 ||r(i)||/||b|| 3.971292601945e-10
  37 KSP preconditioned resid norm 9.733304063786e-29 true resid norm 
1.181603836762e-25 ||r(i)||/||b|| 2.308581589908e-10
  38 KSP preconditioned resid norm 5.660673416565e-29 true resid norm 
6.869825577357e-26 ||r(i)||/||b|| 1.342205598894e-10
  39 KSP preconditioned resid norm 3.306211708906e-29 true resid norm 
3.989295501261e-26 ||r(i)||/||b|| 7.794164054301e-11
  40 KSP preconditioned resid norm 1.970150128293e-29 true resid norm 
2.380707087261e-26 ||r(i)||/||b|| 4.651353001422e-11
  41 KSP preconditioned resid norm 1.149451838466e-29 true resid norm 
1.397019485135e-26 ||r(i)||/||b|| 2.729454123104e-11
  42 KSP preconditioned resid norm 6.718965979215e-30 true resid norm 
8.177436886673e-27 ||r(i)||/||b|| 1.597682714110e-11
  43 KSP preconditioned resid norm 3.957589200762e-30 true resid norm 
4.776461540741e-27 ||r(i)||/||b|| 9.332105088686e-12
  44 KSP preconditioned resid norm 2.316620360281e-30 true resid norm 
2.811297369887e-27 ||r(i)||/||b|| 5.492627181766e-12
  45 KSP preconditioned resid norm 1.358656690701e-30 true resid norm 
1.641732613670e-27 ||r(i)||/||b|| 3.207567180771e-12
  46 KSP preconditioned resid norm 7.970303129905e-31 true resid norm 
9.618857429636e-28 ||r(i)||/||b|| 1.879303069873e-12
  47 KSP preconditioned resid norm 4.693064137122e-31 true resid norm 
5.645411528053e-28 ||r(i)||/||b|| 1.102983310957e-12

But just switching to one process (-n 1) makes it crashing:

[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC 
ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
find memory corruption errors
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames 
------------------------------------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: [0] VecNorm_Seq line 236 
/home/lib/petsc-dev/src/vec/vec/impls/seq/bvec2.c
[0]PETSC ERROR: [0] VecNorm line 150 
/home/lib/petsc-dev/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: [0] VecNormalize line 257 
/home/lib/petsc-dev/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: [0] GMREScycle line 127 
/home/lib/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: [0] KSPSolve_GMRES line 220 
/home/lib/petsc-dev/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: [0] KSPSolve line 334 
/home/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: --------------------- Error Message 
------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development HG revision: 
199bab0ea052fc92ce8e4abb56afc442629a19c8  HG Date: Tue Dec 13 22:22:13 
2011 -0800
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: /home/mt-soft/solveTest/solveTest on a openmpi-i named 
node210 by agrayver Mon Jan  9 12:19:49 2012
[0]PETSC ERROR: Libraries linked from 
/home/lib/petsc-dev/openmpi-intel-complex-debug-f-ds/lib
[0]PETSC ERROR: Configure run at Wed Dec 14 09:49:43 2011
[0]PETSC ERROR: Configure options 
--with-petsc-arch=openmpi-intel-complex-debug-f-ds 
--with-fortran-interfaces=1 --download-superlu --download-superlu_dist 
--download-mumps --download-pastix --download-parmetis 
--download-ptscotch --download-metis 
--with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a 
--with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include 
--with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a 
--with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include 
--with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex 
--with-blas-lapack-dir=/opt/intel/Compiler/11.1/072/mkl/lib/em64t 
--with-precision=double --with-x=0
[0]PETSC ERROR: 
------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory 
unknown file
--------------------------------------------------------------------------

Who did I do wrong?

Regards,
Alexander

More information about the petsc-dev mailing list