[petsc-users] Do more solver iterations = more communication?

[Barry Smith]

> On Feb 18, 2016, at 1:56 PM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> Hi all,
> 
> For a poisson problem with roughly 1 million dofs (using second-order elements), I solved the problem using two different solver/preconditioner combinations: CG/ILU and CG/GAMG. 
> 
> ILU takes roughly 82 solver iterations whereas with GAMG it takes 14 iterations (wall clock time is roughly 15 and 46 seconds respectively). I have seen from previous mailing threads that there is a strong correlation between solver iterations and communication (which could lead to less strong-scaling scalability). It makes sense to me if I strictly use one of these preconditioners to solve two different problems and compare the number of respective iterations, but what about solving the same problem with two different preconditioners?
> 
> If GAMG takes 14 iterations whereas ILU takes 82 iterations, does this necessarily mean GAMG has less communication?

  No you can't say that at all. A single GAMG cycle will do more communication than a single block Jacobi cycle.

> I would think that the "bandwidth" that happens within a single GAMG iteration would be much greater than that within a single ILU iteration. Is there a way to officially determine this?
> 
> I see from log_summary that we have this information:
> MPI Messages:         5.000e+00      1.00000   5.000e+00  5.000e+00
> MPI Message Lengths:  5.816e+07      1.00000   1.163e+07  5.816e+07
> MPI Reductions:       2.000e+01      1.00000
> 
> Can this information be used to determine the "bandwidth"?

   You can certainly use this data for each run to determine which algorithm is sending more messages, total length of messages is bigger etc. And if you divided by time it would tell the rate of communication for the different algorithms.

Note that counts of messages and lengths are also given in the detailed table for each operation.

There are also theoretical bounds on messages that can be derived for some iterations applied to some problems.

> If so, does PETSc have the ability to document this for other preconditioner packages like HYPRE's BoomerAMG or Trilinos' ML?

   No, because they don't log this information.
> 
> Thanks,
> Justin