[petsc-users] Software for load balancing to improve parallel performance to be used with PETSc

[Barry Smith]

On Jan 3, 2012, at 5:03 PM, Jed Brown wrote:

> On Tue, Jan 3, 2012 at 16:58, TAY wee-beng <zonexo at gmail.com> wrote:
> I'm running a 3D Fortran CFD code. The grid used is Cartesian. The current code is partitioned in the z direction for MPI.
> 
> For e.g. for total size z = 10, if partitioned into 5 cpus, it'll become size z = 2 for each cpu.
> 
> Uneven grids are used to reduce the number of grids and the main bulk of grids clusters around the center.
> 
> I read about load balancing software. I wonder if it will improve the performance/speed of my code.
> 
> If so, what are the available choices for use with PETSc and Fortran? Are ParMETIS, Zoltan or Isorropi recommended?
> 
> I would just use MatPartitioning (usually calling into ParMetis underneath) if you want an unstructured partition. Zoltan (and its more C++/Epetra-ified Isorropia interface) provides some assistance for moving application data, but I haven't found it to be easier to use than just moving the data myself and it adds an additional dependency.


   Huh? Since it is a structured cartesian mesh code you just want to split up the z direction so that each process has an equal number of grid points (which likely you are already doing, suddenly introducing unstructured partitioning on top of this seems insane). Then when you run with -log_summary you can see the load balance in work (flops) and time for each part of the computation and determine if they are close to being equal. It's crazy to do a done of code development without knowing if load balancing is the problem. Also if you have only 2 sets of z direction values per process you are going to be doing way to much communication relative to the computation. Why not use 3d decomposition by slicing cleanly in all three directions, the end result will require much less communication.


   Barry