[petsc-dev] Hybrid MPI/OpenMP reflections

[Karl Rupp]

Hi Michael,

 > We have recently been trying to re-align our OpenMP fork
> (https://bitbucket.org/ggorman/petsc-3.3-omp) with petsc/master. Much of
> our early work has now been superseded by the threadcomm
> implementations. Nevertheless, there are still a few algorithmic
> differences between the two branches:
>
> 1) Enforcing MPI latency hiding by using task-based spMV:
> If the MPI implementation used does not actually provide truly
> asynchronous communication in hardware, performance can be increased by
> dedicating a single thread to overlapping MPI communication in PETSc.
> However, this is arguably a vendor-specific fix which requires
> significant code changes (ie the parallel section needs to be raised up
> by one level). So perhaps the strategy should be to give guilty vendors
> a hard time rather than messing up the current abstraction.

When using good preconditioners, spMV is essentially never the 
bottleneck and hence I don't think a separate communication thread 
should be implemented in PETSc. Instead, such a fallback should be part 
of a good MPI implementation.


> 2) Nonzero-based thread partitioning:
> Rather than evenly dividing the number of rows among threads, we can
> partition the thread ownership ranges according to the number of
> non-zeros in each row. This balances the work load between threads and
> thus increases strong scalability due to optimised bandwidth
> utilisation. In general, this optimisation should integrate well with
> threadcomms, since it only changes the thread ownership ranges, but it
> does require some structural changes since nnz is currently not passed
> to PetscLayoutSetUp. Any thoughts on whether people regard such a scheme
> as useful would be greatly appreciated.

This is a reasonable optimization, I used a similar strategy for sparse 
matrices on the GPU. Others should comment on whether the interface 
change to PetscLayoutSetUp is acceptable.


> 3) MatMult_SeqBAIJ not threaded:
> Is there a reason why MatMult has not been threaded for BAIJ matrices,
> or is somebody already working on this? If not, I would like to prepare
> a pull request for this using the same approach as MatMult_SeqAIJ.

To my knowledge, it 'simply hasn't been implemented yet'. A pull request 
would be nice, I'm happy to review.

Best regards,
Karli