[petsc-users] h-FGMRES

[Barry Smith]

   You should be using PETSc version 3.4 which was recently released and is what the paper is based on.

    Barry

On May 19, 2013, at 10:11 PM, Gaetan Kenway <gaetank at gmail.com> wrote:

> Hi Everyone
> 
> I am trying to replicate the type of preconditioner described in "Hierarchical and Nested Krylov Methods for Extreme-Scale Computing". 
> 
> I have used the following options: (I'm using fortran so the following is my petsc_options file)
> 
> # Matrix Options
> -matload_block_size 5
> -mat_type mpibaij
> 
> # KSP solver options
> -ksp_type gmres
> -ksp_max_it 1000
> -ksp_gmres_restart 200
> -ksp_monitor
> -ksp_view
> -ksp_pc_side right
> -ksp_rtol 1e-6
> 
> # Nested GMRES Options
> -pc_type bjacobi
> -pc_bjacobi_blocks 4
> -sub_ksp_type gmres
> -sub_ksp_max_it 5
> -sub_pc_type bjacobi
> -sub_sub_pc_type ilu
> -sub_sub_pc_factor_mat_ordering_type rcm
> -sub_sub_pc_factor_levels 1
> 
> The test is run on 64 processors and the total  number of block jacobi blocks is 4 (less than nproc). The error I get is: 
> 
> [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [6]PETSC ERROR:       INSTEAD the line number of the start of the function
> [6]PETSC ERROR:       is given.
> [6]PETSC ERROR: [6] PCSetUp_BJacobi_Multiproc line 1269 /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c
> [6]PETSC ERROR: [6] PCSetUp_BJacobi line 24 /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c
> [6]PETSC ERROR: [6] PCSetUp line 810 /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/interface/precon.c
> [6]PETSC ERROR: [6] KSPSetUp line 182 /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: [6] KSPSolve line 351 /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c
> [6]PETSC ERROR: --------------------- Error Message ------------------------------------
> [6]PETSC ERROR: Signal received!
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 5, Sat Dec  1 15:10:41 CST 2012 
> [6]PETSC ERROR: See docs/changes/index.html for recent updates.
> [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: ------------------------------------------------------------------------
> [6]PETSC ERROR: ./main on a intel-rea named gpc-f109n001 by kenway Sun May 19 23:01:52 2013
> [6]PETSC ERROR: Libraries linked from /home/j/jmartins/kenway/packages/petsc-3.3-p5/intel-real-debug/lib
> [6]PETSC ERROR: Configure run at Sun Jan 20 15:52:20 2013
> [6]PETSC ERROR: Configure options --with-shared-libraries --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes --with-fortran-interfaces=1 --with-debugging=yes --with-scalar-type=real -with-petsc-arch=intel-real-debug --with-blas-lapack-dir= --with-pic
> [6]PETSC ERROR: ------------------------------------------------------------------------
> 
> If the number of blocks is greater than or equal to the number of processors it runs fine.  I'm using version 3.3-p5. 
> 
> The options as listed in the paper are:
> -flow_ksp_type fgmres -flow_ksp_pc_side right -flow_pc_type bjacobi -flow_pc_bjacobi_blocks ngp
> -flow_sub_ksp_type gmres -flow_sub_ksp_max_it 6 -flow_sub_pc_type bjacobi
> -flow_sub_sub_pc_type ilu
> 
> Any suggestions would be greatly appreciated. 
> 
> Thank you,
> 
> Gaetan Kenway
> 
>