[petsc-dev] Possible bugs when using TS with ViennaCL (continued)

[Mani Chandra]

Hi Karl,

Thanks for the explanation. Do you think it will help if I use a GPU which
is capable of doing double precision arithmetic?

I am using NVIDIA Quadra FX 1800 M. It has 1GB of global memory.
Unfortunately NVIDIAs' visual profiler does not seem to work with its
OpenCL implementation. The code does not crash when I run it on the CPU
using Intels OpenCL.

I mean't to say that the code does not crash either with
ComputeResidualViennaCL or ComputeResidual with the normal Petsc Vec/Mats
but does indeed crash when either of them are used with the ViennaCL vecs.
Do you think there is memory allocation at every time step? I thought all
the memory would be allocated during initialization.

Cheers,
Mani




On Tue, Jan 28, 2014 at 3:09 AM, Karl Rupp <rupp at mcs.anl.gov> wrote:

> Hi Mani,
>
>
>  I've been testing further, the code using TS with ViennaCL and there are
>> a couple of things I wanted to point out
>>
>> 1) When using the ComputeResidualViennaCL with either the normal Petsc
>> Vecs/Mats or Vec/MatViennaCL, and using the GPU, the nonlinear
>> convergence is very different from using an OpenCL CPU backend or just
>> the regular Petsc code.
>>
>> a) Using NVIDIA OpenCL to run on the GPU to compute the residual and
>> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat:
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 4.789374470711e-01
>>      1 SNES Function norm 5.491749197245e-02
>>      2 SNES Function norm 6.542412564158e-03
>>      3 SNES Function norm 7.800844032317e-04
>>      4 SNES Function norm 9.349243191537e-05
>>      5 SNES Function norm 1.120692741097e-05
>> 1 TS dt 10 time 10
>>
>> b) Using Intel OpenCL to run on the CPU to compute the residual and
>> using either normal Petsc Vec/Mat or ViennaCL Vec/Mat::
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 3.916582465172e-02
>>      1 SNES Function norm 4.990998832000e-07
>>
>> c) Using ComputeResidual (which runs on the CPU) with the normal Petsc
>> Vec/Mat
>>
>> 0 TS dt 10 time 0
>>      0 SNES Function norm 3.916582465172e-02
>>      1 SNES Function norm 4.990998832000e-07
>> 1 TS dt 10 time 10
>>
>> You see that b) and c) match perfectly but a) is quite different. Why
>> could this be?
>>
>
> The reason are different arithmetic units. Your OpenCL kernel contains
> dx_dt[INDEX_GLOBAL(i,j,var)] -
>    (x[INDEX_GLOBAL(i+1,j,var)] -
>     x[INDEX_GLOBAL(i,j,var)])/DX1 -
>    (x[INDEX_GLOBAL(i,j+1,var)] -
>    x[INDEX_GLOBAL(i,j,var)])/DX2
> so you are subtracting values of about the same magnitude multiple times.
> You get consistent results on the CPU because the same arithmetic units get
> used irrespective of OpenCL-based or 'native' execution. The NVIDIA GPU has
> different round-off behavior. You are likely to see similar effects with
> AMD GPUs. There is nothing we can do to change this.
>
>
>
>  2) When I try using either ComputeResidual or ComputeResidualViennaCL
>> with the ViennaCL Vec/Mats, the GPU run crashes at a late time because
>> of a routine in ViennaCL.
>>
>> ViennaCL: FATAL ERROR: Kernel start failed for 'vec_mul'.
>> ViennaCL: Smaller work sizes could not solve the problem.
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Error in external library!
>> [0]PETSC ERROR: ViennaCL error: ViennaCL: FATAL ERROR:
>> CL_MEM_OBJECT_ALLOCATION_FAILURE
>>
>> I have attached the full crash log. The crash occurs late into the run,
>> in this case at the 80th time step. I thought all memory allocation
>> occurs at the beginning of the run, so I don't quite understand why its
>> failing.
>>
>
> Okay, this sounds like the GPU ultimately runs out of memory. Which GPU do
> you use? How much memory does it have? Do you also see an increase in
> memory consumption with the Intel OpenCL SDK?
>
>
>
> > Note that the code works if I use ComputeResidualViennaCL with
> > the normal Petsc Vec/Mats.
>
> You mean ComputeResidual(), don't you?
>
> Best regards,
> Karli
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140128/85a9b5cf/attachment.html>