Petsc solver question

[Matthew Knepley]

On Thu, Sep 18, 2008 at 1:31 PM, Nguyen, Hung V ERDC-ITL-MS
<Hung.V.Nguyen at usace.army.mil> wrote:
> All,
>
> I have a test code that reads matrix in CSR format, rhs and provided
> solution. I run this code on Cray XT3 system with 1, 2, 4, 8 and 16 cores.
> The timing spent on KSP are 53.50, 213.69, 110.43, 66.90, and 35.60 secs for
> 1, 2, 4, 8, and 16 cores; respectively. Why is a running time on 1 core
> smaller than others (2, 4, 8 cores)? Does re-ordering of matrix help? Did I
> do something wrong in the code?

It appears that you are using Block-Jacobi ILU(0). This is a
completely different
preconditioner for each different number of processes.

   Matt

> Thanks for your help.
>
> Note: the matrix, rhs, provided and computed solution are written to matlab
> files via petsc functions. It seems to me that we got the right solution
> (using matlab to verify). Then, Here is some info about matrix A:
>
> hvnguyen:sapphire01% head matrix.m
> % Size = 59409 59409
> % Nonzeros = 1113875
> zzz = zeros(1113875,3);
>
>>> cond (Mat_0)
> Warning: Using CONDEST instead of COND for sparse matrix.
>> In cond at 28
>
> ans =
>
>   3.9310e+09
>
> -- 1 pes:
>  hvnguyen:sapphire09% yod -np 1 ./test_matrix -ksp_type cg -pc_type bjacobi
> -ksp_rtol 1.0e-15 -ksp_max_it 50000
> yod:  -sz is 1, so -proc is reset from VN to 0
>  reading nlrn ...
>  reading lrn ...
>  reading cmatrix and rld ...
>  Number of iterations and Time in PETSc solver =         2603
>    53.50315904617310       secs
>  2 norm of the error from the provided & computed solution =
>   3.2424850631419981E-008
>  Maximum error from the provided & computed solution (infinity norm) =
>   6.1342397827957029E-010
>  1 norm of the error from the provided & computed solution
>   4.8108500117322262E-006
>
> -- 2 pes:
> hvnguyen:sapphire09% yod -np 2 ./test_matrix -ksp_type cg -pc_type bjacobi
> -ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
>  reading lrn ...
>  reading cmatrix and rld ...
>  Number of iterations and Time in PETSc solver =        17934
>    213.6974561214447   secs
>  2 norm of the error from the provided & computed solution =
>   2.4050101709163445E-008
>  Maximum error from the provided & computed solution (infinity norm) =
>   4.5970760531588439E-010
>  1 norm of the error from the provided & computed solution
>   3.0008169642927509E-006
> FORTRAN STOP
>
> -- 4 peshvnguyen:sapphire09% yod -np 4 ./test_matrix -ksp_type cg -pc_type
> bjacobi -ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
>  reading lrn ...
>  reading cmatrix and rld ...
>  Number of iterations and Time in PETSc solver =        19363
>    110.4340059757233       secs
>  2 norm of the error from the provided & computed solution =
>   1.5643187342954139E-008
>  Maximum error from the provided & computed solution (infinity norm) =
>   3.4082958677572606E-010
>  1 norm of the error from the provided & computed solution
>   2.0655232735563973E-006
>
> --- 8 pes
> hvnguyen:sapphire09% yod -np 8 ./test_matrix -ksp_type cg -pc_type bjacobi
> -ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
>  reading lrn ...
>  reading cmatrix and rld ...
>  Number of iterations and Time in PETSc solver =        25953
>    66.90357208251953       secs
>  2 norm of the error from the provided & computed solution =
>   2.0977543976205624E-008
>  Maximum error from the provided & computed solution (infinity norm) =
>   4.0076741925076931E-010
>  1 norm of the error from the provided & computed solution
>   2.6098351291261861E-006
>
> --- 16 pes
> hvnguyen:sapphire09% yod -np 16 ./test_matrix -ksp_type cg -pc_type bjacobi
> -ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
>  reading lrn ...
>  reading cmatrix and rld ...
>  Number of iterations and Time in PETSc solver =        25842
>    35.60693192481995   secs
>  2 norm of the error from the provided & computed solution =
>   1.5121379940662423E-008
>  Maximum error from the provided & computed solution (infinity norm) =
>   3.1954527912603226E-010
>  1 norm of the error from the provided & computed solution
>   2.0324863426877444E-006
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener