Petsc solver question

[Nguyen, Hung V ERDC-ITL-MS]

All,

I have a test code that reads matrix in CSR format, rhs and provided
solution. I run this code on Cray XT3 system with 1, 2, 4, 8 and 16 cores.
The timing spent on KSP are 53.50, 213.69, 110.43, 66.90, and 35.60 secs for
1, 2, 4, 8, and 16 cores; respectively. Why is a running time on 1 core
smaller than others (2, 4, 8 cores)? Does re-ordering of matrix help? Did I
do something wrong in the code? 

Thanks for your help.

Note: the matrix, rhs, provided and computed solution are written to matlab
files via petsc functions. It seems to me that we got the right solution
(using matlab to verify). Then, Here is some info about matrix A:  

hvnguyen:sapphire01% head matrix.m
% Size = 59409 59409
% Nonzeros = 1113875
zzz = zeros(1113875,3);

>> cond (Mat_0)
Warning: Using CONDEST instead of COND for sparse matrix.
> In cond at 28

ans =

   3.9310e+09

-- 1 pes:
 hvnguyen:sapphire09% yod -np 1 ./test_matrix -ksp_type cg -pc_type bjacobi
-ksp_rtol 1.0e-15 -ksp_max_it 50000
yod:  -sz is 1, so -proc is reset from VN to 0
 reading nlrn ...
 reading lrn ...
 reading cmatrix and rld ...
 Number of iterations and Time in PETSc solver =         2603
    53.50315904617310       secs
 2 norm of the error from the provided & computed solution =
   3.2424850631419981E-008
 Maximum error from the provided & computed solution (infinity norm) =
   6.1342397827957029E-010
 1 norm of the error from the provided & computed solution
   4.8108500117322262E-006

-- 2 pes:
hvnguyen:sapphire09% yod -np 2 ./test_matrix -ksp_type cg -pc_type bjacobi
-ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
 reading lrn ...
 reading cmatrix and rld ...
 Number of iterations and Time in PETSc solver =        17934
    213.6974561214447   secs
 2 norm of the error from the provided & computed solution =
   2.4050101709163445E-008
 Maximum error from the provided & computed solution (infinity norm) =
   4.5970760531588439E-010
 1 norm of the error from the provided & computed solution
   3.0008169642927509E-006
FORTRAN STOP

-- 4 peshvnguyen:sapphire09% yod -np 4 ./test_matrix -ksp_type cg -pc_type
bjacobi -ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
 reading lrn ...
 reading cmatrix and rld ...
 Number of iterations and Time in PETSc solver =        19363
    110.4340059757233       secs
 2 norm of the error from the provided & computed solution =
   1.5643187342954139E-008
 Maximum error from the provided & computed solution (infinity norm) =
   3.4082958677572606E-010
 1 norm of the error from the provided & computed solution
   2.0655232735563973E-006

--- 8 pes
hvnguyen:sapphire09% yod -np 8 ./test_matrix -ksp_type cg -pc_type bjacobi
-ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
 reading lrn ...
 reading cmatrix and rld ...
 Number of iterations and Time in PETSc solver =        25953
    66.90357208251953       secs
 2 norm of the error from the provided & computed solution =
   2.0977543976205624E-008
 Maximum error from the provided & computed solution (infinity norm) =
   4.0076741925076931E-010
 1 norm of the error from the provided & computed solution
   2.6098351291261861E-006

--- 16 pes
hvnguyen:sapphire09% yod -np 16 ./test_matrix -ksp_type cg -pc_type bjacobi
-ksp_rtol 1.0e-15 -ksp_max_it 50000  reading nlrn ...
 reading lrn ...
 reading cmatrix and rld ...
 Number of iterations and Time in PETSc solver =        25842
    35.60693192481995   secs
 2 norm of the error from the provided & computed solution =
   1.5121379940662423E-008
 Maximum error from the provided & computed solution (infinity norm) =
   3.1954527912603226E-010
 1 norm of the error from the provided & computed solution
   2.0324863426877444E-006
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_matrix.F
Type: application/octet-stream
Size: 5185 bytes
Desc: test_matrix.F
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20080918/a4cf9928/attachment.obj>