[petsc-dev] Supporting OpenCL matrix assembly

[Jed Brown]

Karl Rupp <rupp at mcs.anl.gov> writes:

> Hey,
>
>> Perhaps that *GetSource method should also return an opaque device "Mat"
>> pointer that the user is responsible for shepherding into the kernel
>>  From which they call the device MatSetValues?
>
> This is easy of the OpenCL management is within PETSc (i.e. context, 
> buffers and command queues managed by us). I expect that a bunch of 
> users wants to provide their own context and stuff, which would require 
> us to offer something like
>    MatAttachOpenCLEnvironment(Mat,cl_context,cl_command_queue);
> for all the matrix and vector objects involved. Note that this needs to 
> be attached before the matrix is created. I think this is doable.

Okay, but why do they need to provide their own "Mat" data?

I envision the user doing all the job launching so they have control
over everything.  The Mat "implements" its MatSetValuesOpenCL (called on
the device) so it needs to provide the device handle.

What am I missing?

>> Suppose the column indices have been set in advance.  Now if the
>> application already has a way of preventing conflicted cross-threadblock
>> writes to those slots within an insertion round (e.g., coloring), PETSc
>> would not need any synchronization and wouldn't need to stash
>> possibly-conflicted writes elsewhere.  Otherwise, PETSc would have to
>> manage the stashing, use atomics, or some other scheme.
>
> I see. My experience is that synchronizations, particularly atomics, are 
> usually too expensive on GPUs if one wants to compete with an optimized 
> CPU implimentation. Coloring is often reasonable, but the price to pay 
> are bad strong scaling properties, because each color induces a ~10us 
> kernel launch overhead. Either way, that's a reasonable implementation 
> approach to start with.

If we only had to color the interfaces between thread blocks, we should
have fewer colors.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130924/6ddcf501/attachment.sig>