[petsc-users] performance of hmpi

[Barry Smith]

On Sep 5, 2012, at 6:27 PM, George Pau <gpau at lbl.gov> wrote:

> Hi,
> 
> I am comparing between 
> 
> mpiexec -n 2 myexec -pc_type bjacobi -ksp_type preonly 

      This "solver" won't work. It does only one iteration of bjacobi and so won't solve the system.  You should use it with -ksp_type gmres (for example)

> 
> and 
> 
> mpiexec -n 1 myexec -pc_type hmpi -ksp_type preonly -hmpi_pc_type bjacobi -hmpi_ksp_type preonly -hmpi_spawn_size 2 
> 
> My matrix size is 10500x10500 and it is a non-symmetric and not necessarily positive definite all the time.    I am not seeing any improvement in performance in hmpi case going from 1 process to 2 processes (in fact there is a slight increase).  With the bjacobi I do see some improvement.  Is this expected due to perhaps communication costs?  Under what situation will hmpi be useful?  Should I be using hmpi only on much larger problems and with more processes?

   This is a small problem to run on two processes. You need larger problems to see a good improvement and it depends a lot on the machine as well. http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

   Since the solver used in your two examples above (if you switch to gmres) is the same and they both run on two processes the solution time should be pretty similar. But the second case has an additional cost of reorganizing the sparse matrix from one process to two so will always be a bit slower. 

   The hmpi is intended for when one's original code is either not parallel (or uses OpenMP) but allows the PETSc solver part of the code to be parallel (with MPI) so you can't really compare the two cases above. You should compare the -hmpi_spawn_size 2 case with the one MPI process case (and then go up to 3 and 4 processes with the spawn).

    You should also use -log_summary to get timing information for each part of the solution process so you can see where the time is being spent.

   Barry

> 
> Thanks,
> George
> 
> -- 
> George Pau
> Earth Sciences Division
> Lawrence Berkeley National Laboratory
> One Cyclotron, MS 74-120
> Berkeley, CA 94720
> 
> (510) 486-7196
> gpau at lbl.gov
> http://esd.lbl.gov/about/staff/georgepau/
>