[petsc-users] performance issue solving multiple linear systems of the same size with the different preconditioning methods

[Jed Brown]

On Sun, Aug 21, 2011 at 16:45, Алексей Рязанов <ram at ibrae.ac.ru> wrote:

> Hello!
>
> Could you please help me to solve my performance problem.
> I have two programs.
>
> In 1st I solve one system with one method and one preconditioner and get
> some performance numbers.
> I run it 9 times with 9 different preconditioners.
>
> In 2nd I solve the same system with the same one method but with 9
> different preconditioners consecutively one after another.
> I run it once and also get some performance info.
> In the 2nd case I have 2-5 times worse results, depending on used method.
>
> Each KSPSolve procedure placed in its own stage of course, so I can compare
> times, flops, messages an so..
> I can see the difference but cant explain and eliminate it.
>
> For example for -ksp_type cgs -pc_type asm -sub_pc_type jacobi
> -sub_ksp_type preonly:
> Summary of Stages:        ----- Time ------                 ----- Flops
> -----       --- Messages ---  -- Message Lengths --  -- Reductions --
>                                             Avg     %Total       Avg
> %Total      counts   %Total        Avg         %Total   counts   %Total
>     one stage frome 2nd:   5.5145e+00  14.9%  1.2336e+09  13.2%  2.494e+03
>  22.4%  3.230e+03       22.2%  2.250e+03  18.5%
> the once stage from 1st:  2.7541e+00  93.1%  1.2336e+09  99.8%  2.508e+03
>  99.3%  1.470e+04       97.4%  2.266e+03  88.0%
>
>
> My programs are pretty equivalent except the part with definition of
> preconditioners and the number of called KSPSolve procedures.
> I mean they use equal matrices, equally assemble them, use equal right hand
> sides, equal convergence monitors.
> Actually the 2nd one was made from the 1st.
>
> In 1st i use KSPSetFromOptions(KSP); and then just set the -ksp_type
>  -pc_type -sub_pc_type -sub_ksp_type keys from command line
>
> In 2d i use for for nonblock PC:
>
>   KSPGetPC(dKSP, &dPC);
>
>   PCSetType(dPC, PCJACOBI);
>
> and for block PC:
>
>   PCSetType(dPC, PCASM);
>
>   KSPSetUp(dKSP);
>
>   PCSetUp(dPC);
>
>   PCASMGetSubKSP(dPC, &n_local, &first_local, &ASMSubKSP);
>
>   for (i=0; i<n_local; i++)
>
>   {
>
>     KSPGetPC(ASMSubKSP[i], &(SubPC[i]));
>
>     PCSetType(SubPC[i], PCJACOBI);
>
>   }
>
>
> Im sure there is a mistake somewhere. Because 1st program compares Jacobi
> and ASM-Jacobi preconditioners on my problem on the same KSP and tells me
> that ASM-Jacobi is better and the 2nd shows otherwise results.
>

This could be a preload issue. You can use the PreLoadBegin()/PreLoadEnd()
macros if you like, or otherwise solve a system first to make sure
everything has been loaded. If the results are still confusing, run with
-ksp_view -log_summary and send the output.

There is no reason for ASM-Jacobi (with -sub_ksp_type preonly, which is
default) to be better than Jacobi since it does the same algorithm with more
communication.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110821/513c90cb/attachment.htm>