Question with pc_type redundant

[Hong Zhang]

Hung,

PC redundant is used for solving coarse-grid linear problem,
for which the problem is too small for parallel processing.
Thus, we assign each(default) or a subset of procesors 
to solve the problem redundantly with LU direct solver.
In your case, it is likely all processors implement
LU sequentially. Adding the time of broadcasting matrix
entries to all processors, more processors take longer time
is anticipated.

Hong

On Thu, 20 Nov 2008, Nguyen, Hung V ERDC-ITL-MS wrote:

> All,
>
> I try to test the pc_type of redundant with ksp_type of stcg and cg. For both
> cases, why does the elapsed times are large when running with large number of
> processors?
>
> Thanks for your help.
>
> -Hung
>
> hvnguyen:jade17% head A.m
> % Size = 59409 59409
> % Nonzeros = 1113875
> zzz = zeros(1113875,3);
>
> 1. Test with stcg and redundant:
>
> hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.117929341660e-06
>  0 KSP Residual norm 2.394961013957e-16
> Time in PETSc solver: 26.436834 seconds
> The number of iteration       = 2
> The solution residual error = 2.394961e-16
> 2 norm 1.848728e-08
> infinity norm 4.014424e-10
> 1 norm 2.288013e-06
>
> hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.108986752645e-06
>  0 KSP Residual norm 2.388254585478e-16
> Time in PETSc solver: 28.498422 seconds
> The number of iteration       = 2
> The solution residual error = 2.388255e-16
> 2 norm 1.763764e-08
> infinity norm 3.500773e-10
> 1 norm 2.543252e-06
>
> hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.076617230240e-06
>  0 KSP Residual norm 2.364985060528e-16
> Time in PETSc solver: 34.782896 seconds
> The number of iteration       = 2
> The solution residual error = 2.364985e-16
> 2 norm 3.544232e-08
> infinity norm 4.848555e-10
> 1 norm 5.478173e-06
>
> hvnguyen:jade17% setenv MPICH_UNEX_BUFFER_SIZE 100M
> hvnguyen:jade17% setenv      MPICH_PTL_MATCH_OFF    1
> hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.076187797630e-06
>  0 KSP Residual norm 2.327317778542e-16
> Time in PETSc solver: 34.810215 seconds
> The number of iteration       = 2
> The solution residual error = 2.327318e-16
> 2 norm 3.448040e-08
> infinity norm 4.582703e-10
> 1 norm 5.306073e-06
>
> hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.170520566185e-06
>  0 KSP Residual norm 2.480424058413e-16
> Time in PETSc solver: 34.851701 seconds
> The number of iteration       = 2
> The solution residual error = 2.480424e-16
> 2 norm 6.318255e-08
> infinity norm 8.093555e-10
> 1 norm 9.681583e-06
>
>
> hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type stcg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  0 KSP Residual norm 1.072600099318e-06
>  0 KSP Residual norm 2.340608377807e-16
> Time in PETSc solver: 34.951315 seconds
> The number of iteration       = 2
> The solution residual error = 2.340608e-16
> 2 norm 3.767959e-08
> infinity norm 5.213003e-10
> 1 norm 5.790317e-06
>
>
> 2. Test with cg and redundant:
>
> hvnguyen:jade17% aprun -n 1 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.117929341660e-06
>  2 KSP Residual norm 2.394961013957e-16
> Time in PETSc solver: 26.460811 seconds
> The number of iteration       = 2
> The solution residual error = 2.394961e-16
> 2 norm 1.848728e-08
> infinity norm 4.014424e-10
> 1 norm 2.288013e-06
>
> hvnguyen:jade17% aprun -n 2 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.108986752645e-06
>  2 KSP Residual norm 2.388254585478e-16
> Time in PETSc solver: 28.494262 seconds
> The number of iteration       = 2
> The solution residual error = 2.388255e-16
> 2 norm 1.763764e-08
> infinity norm 3.500773e-10
> 1 norm 2.543252e-06
>
> hvnguyen:jade17% aprun -n 4 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.076617230240e-06
>  2 KSP Residual norm 2.364985060528e-16
> Time in PETSc solver: 34.746018 seconds
> The number of iteration       = 2
> The solution residual error = 2.364985e-16
> 2 norm 3.544232e-08
> infinity norm 4.848555e-10
> 1 norm 5.478173e-06
>
> hvnguyen:jade17% aprun -n 8 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.076187797630e-06
>  2 KSP Residual norm 2.327317778542e-16
> Time in PETSc solver: 34.830512 seconds
> The number of iteration       = 2
> The solution residual error = 2.327318e-16
> 2 norm 3.448040e-08
> infinity norm 4.582703e-10
> 1 norm 5.306073e-06
>
> hvnguyen:jade17% aprun -n 16 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.170520566185e-06
>  2 KSP Residual norm 2.480424058413e-16
> Time in PETSc solver: 34.866815 seconds
> The number of iteration       = 2
> The solution residual error = 2.480424e-16
> 2 norm 6.318255e-08
> infinity norm 8.093555e-10
> 1 norm 9.681583e-06
>
> hvnguyen:jade17% aprun -n 32 ./test_matrix_read -ksp_type cg -pc_type
> redundant -ksp_rtol 1.0e-12 -ksp_max_it 50000 -ksp_monitor
>  0 KSP Residual norm 1.573381744277e+04
>  1 KSP Residual norm 1.072600099318e-06
>  2 KSP Residual norm 2.340608377807e-16
> Time in PETSc solver: 34.968400 seconds
> The number of iteration       = 2
> The solution residual error = 2.340608e-16
> 2 norm 3.767959e-08
> infinity norm 5.213003e-10
> 1 norm 5.790317e-06
>
>
>