[petsc-users] Tuning the parallel performance of a 3D FEM CFD code

[Barry Smith]

On May 12, 2011, at 8:41 AM, Henning Sauerland wrote:

> Hi, 
> 
> I'm using PETSC with a finite element incompressible two-phase Navier-Stokes code. In 3D I'm currently struggling with the parallel performance. The domain partitioning is done with ParMeTis and PETSC is used for matrix assembly and solving. Unfortunately, for this problem the number of processors has a large influence on the number of iterations leading to a poor scaling. I have already tested a lot of solver-preconditioner combinations. LGMRES with ASM preconditioning seems to perform best. Applying -sub_pc_type lu helped a lot in 2D, but in 3D apart from reducing the number of iterations

   Exactly how much did it reduce the number of iterations? For example on 4 processes it is taking 500 iterations, with LU instead how many iterations does it take? 50? 100? 400? 

   Also how many iterations does ILU need on one process? 

    We have to get that iteration count down dramatically, to do this we need to start by understanding relatively how much the ILU is hurting you versus the overlapping  Schwarz decomposition. Do you some kind of saddle-pioint thingy in your Jacobian? (if so I'm guessing you are not doing anything to deal with it). 

  Barry


> the whole solution takes more than 10 times longer. I attached the log_summary output for a problem with about 240000 unkowns (1 time step) using 4, 8 and 16 Intel Xeon E5450 processors (InfiniBand-connected). As far as I see the number of iterations seems to be the major issue here or am I missing something? I would appreciate any suggestions as to what I could try to improve the scaling.
> 
> 
> Thanks
> Henning
> 
> 
> <4pe.txt><8pe.txt><16pe.txt>
> 
>