[petsc-users] Interations increase with parallelism

[Matthew Knepley]

On Tue, May 1, 2012 at 6:20 PM, <tibo at berkeley.edu> wrote:

> Dear petsc users,
>
> I am solving large systems of non linear PDEs. To do so, the most
> expensive operation is to solve linear systems Ax=b, where A is block
> tridiagonal. To do so, I am using petsc.
>
> A is created using MatCreateMPIAIJ and x and b using VecCreateMPI, and I
> do not modify default parameters of the KSP context for now (ie the Krylov
> method is GMRES and the preconditioner method is ILU(0) if I use 1
> processor - sequential matrix - and Block Jacobi with ILU(0) for each
> sub-block if I use more than 1 processor).
>
> For any number n of processors used, I do get the correct result. However,
> it seems that the more processors I have, the more iterations are done on
> each linear solve (n = 1 gives about 1-2 iterations per solve, n = 2 gives
> 8-12 iterations per solve, n = 4 gives 15-20 iterations per solve).
>
> While I can understand the difference between n = 1 and n = 2, since the
> preconditioning method changes from ILU(0) to Block Jacobi, I don't
> understand why this is the case from n = 2 to 4 for example, since it
> seems to me that the method used to solve Ax=b will be the same (although
> the partitioning is different) and so the operations will be the same,
> even though there will be more communication.
>
> My first question is then: Is this normal behavior or am I probably wrong
> somewhere ?
>

Its not the same from 2-4. You have 4 smaller blocks in BJACOBI and less
coupling.

Also, since the increase in the number of iterations more than offsets the
> decrease in time spent solving the system when n increase, my program runs
> slower with an increasing number of processors, which is the opposite of
> what is desired...Would you have suggestions as what I could change to
> correct this ?
>

Scalable preconditioning is a research topic. However, the next thing to
try is AMG.


> I would be happy to provide more details about the problem/datastructures
> used if needed,
>

The best thing to try usually is a literature search for people using
scalable preconditioners
for your kind of physics.

    Matt


> thank you for your help,
>
> Tibo
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120501/e244893b/attachment-0001.htm>