Question with pc_type redundant

[Nguyen, Hung V ERDC-ITL-MS]

All,

I try to test the pc_type of redundant with ksp_type of stcg and cg. For both
cases, why does the elapsed times are large when running with large number of
processors? 

Thanks for your help.

-Hung

hvnguyen:jade17% head A.m
% Size = 59409 59409
% Nonzeros = 1113875
zzz = zeros(1113875,3);

1. Test with stcg and redundant:

hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.117929341660e-06
  0 KSP Residual norm 2.394961013957e-16
Time in PETSc solver: 26.436834 seconds
The number of iteration       = 2
The solution residual error = 2.394961e-16
2 norm 1.848728e-08
 infinity norm 4.014424e-10
 1 norm 2.288013e-06

hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.108986752645e-06
  0 KSP Residual norm 2.388254585478e-16
Time in PETSc solver: 28.498422 seconds
The number of iteration       = 2
The solution residual error = 2.388255e-16
2 norm 1.763764e-08
 infinity norm 3.500773e-10
 1 norm 2.543252e-06

hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.076617230240e-06
  0 KSP Residual norm 2.364985060528e-16
Time in PETSc solver: 34.782896 seconds
The number of iteration       = 2
The solution residual error = 2.364985e-16
2 norm 3.544232e-08
 infinity norm 4.848555e-10
 1 norm 5.478173e-06

hvnguyen:jade17% setenv MPICH_UNEX_BUFFER_SIZE 100M
hvnguyen:jade17% setenv      MPICH_PTL_MATCH_OFF    1
hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.076187797630e-06
  0 KSP Residual norm 2.327317778542e-16
Time in PETSc solver: 34.810215 seconds
The number of iteration       = 2
The solution residual error = 2.327318e-16
2 norm 3.448040e-08
 infinity norm 4.582703e-10
 1 norm 5.306073e-06

hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.170520566185e-06
  0 KSP Residual norm 2.480424058413e-16
Time in PETSc solver: 34.851701 seconds
The number of iteration       = 2
The solution residual error = 2.480424e-16
2 norm 6.318255e-08
 infinity norm 8.093555e-10
 1 norm 9.681583e-06


hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type stcg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  0 KSP Residual norm 1.072600099318e-06
  0 KSP Residual norm 2.340608377807e-16
Time in PETSc solver: 34.951315 seconds
The number of iteration       = 2
The solution residual error = 2.340608e-16
2 norm 3.767959e-08
 infinity norm 5.213003e-10
 1 norm 5.790317e-06


2. Test with cg and redundant:

hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.117929341660e-06
  2 KSP Residual norm 2.394961013957e-16
Time in PETSc solver: 26.460811 seconds
The number of iteration       = 2
The solution residual error = 2.394961e-16
2 norm 1.848728e-08
 infinity norm 4.014424e-10
 1 norm 2.288013e-06

hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.108986752645e-06
  2 KSP Residual norm 2.388254585478e-16
Time in PETSc solver: 28.494262 seconds
The number of iteration       = 2
The solution residual error = 2.388255e-16
2 norm 1.763764e-08
 infinity norm 3.500773e-10
 1 norm 2.543252e-06

hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.076617230240e-06
  2 KSP Residual norm 2.364985060528e-16
Time in PETSc solver: 34.746018 seconds
The number of iteration       = 2
The solution residual error = 2.364985e-16
2 norm 3.544232e-08
 infinity norm 4.848555e-10
 1 norm 5.478173e-06

hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.076187797630e-06
  2 KSP Residual norm 2.327317778542e-16
Time in PETSc solver: 34.830512 seconds
The number of iteration       = 2
The solution residual error = 2.327318e-16
2 norm 3.448040e-08
 infinity norm 4.582703e-10
 1 norm 5.306073e-06

hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.170520566185e-06
  2 KSP Residual norm 2.480424058413e-16
Time in PETSc solver: 34.866815 seconds
The number of iteration       = 2
The solution residual error = 2.480424e-16
2 norm 6.318255e-08
 infinity norm 8.093555e-10
 1 norm 9.681583e-06

hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type cg -pc_type
redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
  0 KSP Residual norm 1.573381744277e+04
  1 KSP Residual norm 1.072600099318e-06
  2 KSP Residual norm 2.340608377807e-16
Time in PETSc solver: 34.968400 seconds
The number of iteration       = 2
The solution residual error = 2.340608e-16
2 norm 3.767959e-08
 infinity norm 5.213003e-10
 1 norm 5.790317e-06