Problems porting code to an IBM power5+ machine

[Barry Smith]

  On the power5 if the libraries are compiled with --with-debugging=0
build another set (different PETSC_ARCH) with --with-debugging=1
and see what happens. IBM's optimizers are very buggy.

   Barry


On Mon, 19 Mar 2007, Knut Erik Teigen wrote:

> Hello
> 
> I have got a working code on my local machine( Pentium 4), but when I
> try to run the code on a power5+ machine, the equation solver won't
> converge. It seems like it calculates the wrong right hand side norm.
> Below is the result with run-time options 
> "-ksp_type cg -pc_type icc -ksp_monitor -ksp_view -info
> First with the code running on the power5+ machine, then on my local
> machine. I've also printed the right hand side, jacobian matrix and
> solution for a small 3x3 grid.
> 
> Can anyone help me figure out what's wrong?
> 
> Regards,
> Knut Erik Teigen
> 
> Code running on Power5+:
> rhs:
> 980
> 980
> 980
> -0
> -0
> -0
> -980
> -980
> -980
> jacobian:
> row 0: (0, 20)  (1, -10)  (3, -10)
> row 1: (0, -10)  (1, 30)  (2, -10)  (4, -10)
> row 2: (1, -10)  (2, 20)  (5, -10)
> row 3: (0, -10)  (3, 30)  (4, -10)  (6, -10)
> row 4: (1, -10)  (3, -10)  (4, 40)  (5, -10)  (7, -10)
> row 5: (2, -10)  (4, -10)  (5, 30)  (8, -10)
> row 6: (3, -10)  (6, 20)  (7, -10)
> row 7: (4, -10)  (6, -10)  (7, 30)  (8, -10)
> row 8: (5, -10)  (7, -10)  (8, 20)
> [0] PCSetUpSetting up new PC
> [0] PetscCommDuplicateDuplicating a communicator 1 4 max tags =
> 1073741823
> [0] PetscCommDuplicateUsing internal PETSc communicator 1 4
> [0] PetscCommDuplicateUsing internal PETSc communicator 1 4
>   0 KSP Residual norm 7.410163701832e+28
>   1 KSP Residual norm 6.464393707520e+11
> [0] KSPDefaultConvergedLinear solver has converged. Residual norm
> 6.46439e+11 is less than relative tolerance 1e-07 times initial right
> hand side norm 7.41016e+28 at iteration 1
> KSP Object:
>   type: cg
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-07, absolute=1e-50, divergence=10000
>   left preconditioning
> PC Object:
>   type: icc
>     ICC: 0 levels of fill
>     ICC: factor fill ratio allocated 1
>     ICC: factor fill ratio needed 0.636364
>          Factored matrix follows
>         Matrix Object:
>           type=seqsbaij, rows=9, cols=9
>           total: nonzeros=21, allocated nonzeros=21
>               block size is 1
>   linear system matrix = precond matrix:
>   Matrix Object:
>     type=seqaij, rows=9, cols=9
>     total: nonzeros=33, allocated nonzeros=45
>       not using I-node routines
> solution:
> 6.37205e-09
> 7.13167e-09
> 7.49911e-09
> -2.48277e-09
> -4.56885e-10
> 0
> 0
> 0
> 0
> 
> Code running on local machine:
> rhs:
> 980
> 980
> 980
> -0
> -0
> -0
> -980
> -980
> -980
> jacobian:
> row 0: (0, 20)  (1, -10)  (3, -10)
> row 1: (0, -10)  (1, 30)  (2, -10)  (4, -10)
> row 2: (1, -10)  (2, 20)  (5, -10)
> row 3: (0, -10)  (3, 30)  (4, -10)  (6, -10)
> row 4: (1, -10)  (3, -10)  (4, 40)  (5, -10)  (7, -10)
> row 5: (2, -10)  (4, -10)  (5, 30)  (8, -10)
> row 6: (3, -10)  (6, 20)  (7, -10)
> row 7: (4, -10)  (6, -10)  (7, 30)  (8, -10)
> row 8: (5, -10)  (7, -10)  (8, 20)
> [0] PCSetUp(): Setting up new PC
> [0] PetscCommDuplicate(): Duplicating a communicator 1140850689
> -2080374783 max tags = 2147483647
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374783
> [0] PetscCommDuplicate(): Using internal PETSc communicator 1140850689
> -2080374783
>   0 KSP Residual norm 2.505507810276e+02
>   1 KSP Residual norm 3.596555656581e+01
>   2 KSP Residual norm 2.632672485513e+00
>   3 KSP Residual norm 1.888285055287e-01
>   4 KSP Residual norm 7.029433008806e-03
>   5 KSP Residual norm 3.635267067420e-14
> [0] KSPDefaultConverged(): Linear solver has converged. Residual norm
> 3.63527e-14 is less than relative tolerance 1e-07 times initial right
> hand side norm 250.551 at iteration 5
> KSP Object:
>   type: cg
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-07, absolute=1e-50, divergence=10000
>   left preconditioning
> PC Object:
>   type: icc
>     ICC: 0 levels of fill
>     ICC: factor fill ratio allocated 1
>     ICC: factor fill ratio needed 0.636364
>          Factored matrix follows
>         Matrix Object:
>           type=seqsbaij, rows=9, cols=9
>           total: nonzeros=21, allocated nonzeros=21
>               block size is 1
>   linear system matrix = precond matrix:
>   Matrix Object:
>     type=seqaij, rows=9, cols=9
>     total: nonzeros=33, allocated nonzeros=45
>       not using I-node routines
> solution:
> 92.3023
> 92.3023
> 92.3023
> -5.69767
> -5.69767
> -5.69767
> -103.698
> -103.698
> -103.698
> 
> 
> 
>