[petsc-users] h-FGMRES

[Gaetan Kenway]

Hi Everyone

I am trying to replicate the type of preconditioner described in
"Hierarchical and Nested Krylov Methods for Extreme-Scale Computing".

I have used the following options: (I'm using fortran so the following is
my petsc_options file)

# Matrix Options
-matload_block_size 5
-mat_type mpibaij

# KSP solver options
-ksp_type gmres
-ksp_max_it 1000
-ksp_gmres_restart 200
-ksp_monitor
-ksp_view
-ksp_pc_side right
-ksp_rtol 1e-6

# Nested GMRES Options
-pc_type bjacobi
-pc_bjacobi_blocks 4
-sub_ksp_type gmres
-sub_ksp_max_it 5
-sub_pc_type bjacobi
-sub_sub_pc_type ilu
-sub_sub_pc_factor_mat_ordering_type rcm
-sub_sub_pc_factor_levels 1

The test is run on 64 processors and the total  number of block jacobi
blocks is 4 (less than nproc). The error I get is:

[6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[6]PETSC ERROR:       INSTEAD the line number of the start of the function
[6]PETSC ERROR:       is given.
[6]PETSC ERROR: [6] PCSetUp_BJacobi_Multiproc line 1269
/home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c
[6]PETSC ERROR: [6] PCSetUp_BJacobi line 24
/home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c
[6]PETSC ERROR: [6] PCSetUp line 810
/home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/interface/precon.c
[6]PETSC ERROR: [6] KSPSetUp line 182
/home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: [6] KSPSolve line 351
/home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c
[6]PETSC ERROR: --------------------- Error Message
------------------------------------
[6]PETSC ERROR: Signal received!
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41
CST 2012
[6]PETSC ERROR: See docs/changes/index.html for recent updates.
[6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR:
------------------------------------------------------------------------
[6]PETSC ERROR: ./main on a intel-rea named gpc-f109n001 by kenway Sun May
19 23:01:52 2013
[6]PETSC ERROR: Libraries linked from
/home/j/jmartins/kenway/packages/petsc-3.3-p5/intel-real-debug/lib
[6]PETSC ERROR: Configure run at Sun Jan 20 15:52:20 2013
[6]PETSC ERROR: Configure options --with-shared-libraries
--download-superlu_dist=yes --download-parmetis=yes --download-metis=yes
--with-fortran-interfaces=1 --with-debugging=yes --with-scalar-type=real
-with-petsc-arch=intel-real-debug --with-blas-lapack-dir= --with-pic
[6]PETSC ERROR:
------------------------------------------------------------------------

If the number of blocks is greater than or equal to the number of
processors it runs fine.  I'm using version 3.3-p5.

The options as listed in the paper are:
-flow_ksp_type fgmres -flow_ksp_pc_side right -flow_pc_type bjacobi
-flow_pc_bjacobi_blocks ngp
-flow_sub_ksp_type gmres -flow_sub_ksp_max_it 6 -flow_sub_pc_type bjacobi
-flow_sub_sub_pc_type ilu

Any suggestions would be greatly appreciated.

Thank you,

Gaetan Kenway
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130519/62f49290/attachment.html>