[petsc-dev] VecScatterInitializeForGPU

[Paul Mullowney]

Dominic,

A few years ago, I was trying to minimize the amount of data transfer to
and from the GPU (for multi-GPU MatMult) by inspecting the indices of the
data that needed to be message to and from the device. Then, I would call
gather kernels on the GPU which pulled the scattered data into contiguous
buffers and then be transferred to the host asynchronously (while the
MatMult was occurring). The existence of VecScatterInitializeForGPU was
added in order to build the necessary buffers as needed. This was the
motivation behind the existence of VecScatterInitializeForGPU.

An alternative approach is to message the smallest contiguous buffer
containing all the data with a single cudaMemcpyAsync. This is the method
currently implemented.

I never found a case where the former implementation (with a GPU
gather-kernel) performed better than the alternative approach which
messaged the smallest contiguous buffer. I looked at many, many matrices.

Now, as far as I understand the VecScatter kernels, this method should only
get called if the transfer is MPI_General (i.e. PtoP parallel to parallel).
Other VecScatter methods are called in other circumstances where the the
scatter is not MPI_General. That assumption could be wrong though.

-Paul




On Wed, Jan 22, 2014 at 9:49 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:

> Hi,
>
> I'm trying to understand VecScatterInitializeForGPU in
> src/vec/vec/utils/veccusp/vscatcusp.c. I don't understand why this
> function can get away with casting the fromdata and todata in the inctx to
> VecScatter_MPI_General. Don't we need to inspect the VecScatterType fields
> of the todata and fromdata?
>
> Cheers,
> Dominic
>
> --
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140122/8c099db1/attachment.html>