[petsc-dev] cuda failures of tests in master

[Karl Rupp]

Hi Dominic,

 > With the current implementation the following can happen (v is of type
> VECCUSP):
> - Originally data on GPU, v.valid_GPU_array == PETSC_CUSP_GPU
> - a call to VecPlaceArray(v, arr) unplaces the data on the host and sets
> v.valid_CPU_array=CPU. Note that the GPU data does not get stashed.
> - subsequent accesses of the GPU data will clobber the data that was
> there before VecPlaceArray.
>
> I think there are two possible solutions:
> - In VecPlaceArray_SeqCUSP we allocate a new array on the GPU and stash
> the current values.
> - We do a GPU->CPU synchronization in VecPlaceArray_SeqCUSP to make sure
> that the data on the CPU is up to date.
>
> It's a space/time tradeoff. Also, the first option further complicates
> the caching mechanism. I think the caching mechanism is already too
> complicated (nearly every bug I encounter with the CUDA stuff is related
> to caching). The second option allows us to more easily reuse
> VecPlaceArray_Seq. And we don't have to juggle a GPU unplaced array in
> addition to the host side unplaced array. I'd therefore propose that we
> take the hit of a GPU->CPU data synchronization. Not that this
> synchronization only incurs in a PCIe data transfer if the data on the
> CPU is stale.
>
> But all of this is only needed if the semantics of
> VecPlaceArray/VecResetArray array is to preserve the contents of the
> unplaced array.

any attempt of adding too much cleverness to the caching mechanism will 
(imho) ultimately bring up more troubles of this kind, plus makes the 
code too hard to maintain.

I'll soon get to the integration of ViennaCL 1.7.x features including 
full CUDA/OpenCL/OpenMP switching, requiring us to reconsider the 
semantics of VecPlaceArray() and friends. Up until then I'm afraid that 
I might miss some details, so I'll refrain from recommending one option 
over the other.

Best regards,
Karli



> On 08/07/2015 09:23 PM, Barry Smith wrote:
>>
>>> On Aug 7, 2015, at 7:50 PM, Dominic Meiser <dmeiser at txcorp.com> wrote:
>>>
>>> FYI I've opened a pull request that addresses this issue.
>>>
>>> While going through the code I ran into a question regarding the
>>> semantics of VecPlaceArray and VecResetArray: Is the contents of the
>>> "unplaced" array supposed to be preserved so that the vector is
>>> completely restored upon calling VecResetArray? With the current
>>> implementation of VecPlaceArray_SeqCUSP and VecResetArray_SeqCUSP a
>>> situation can occur where the contents of the unplaced array gets
>>> clobbered. Does this need to be fixed?
>>
>>     Hmm,  I think I always assumed the "unplaced" array was
>> inaccessible during the time it is unplaced (since there is no public
>> pointer to the array), this would mean that the values there shouldn't
>> change. What are the exact details of how they can get changed?
>>
>>     Thanks
>>
>>     Barry
>>
>>>
>>> Cheers,
>>> Dominic
>>>
>>>
>>> On 08/07/2015 02:42 PM, Barry Smith wrote:
>>>>
>>>>    Guardians of CUDA/GPUs
>>>>
>>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/08/06/examples_master_arch-cuda-double_bb-proxy.log
>>>>
>>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/08/06/examples_master_arch-cuda_bb-proxy.log
>>>>
>>>>
>>>> search for ex2_bjacobi
>>>>
>>>> note that this example does not fail in non CUDA builds.
>>>>
>>>> For some reason the iterative solver thinks it converges in 0
>>>> iterations but the answer is completely wrong.
>>>>
>>>>    Barry
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Dominic Meiser
>>> Tech-X Corporation
>>> 5621 Arapahoe Avenue
>>> Boulder, CO 80303
>>> USA
>>> Telephone: 303-996-2036
>>> Fax: 303-448-7756
>>> www.txcorp.com
>>
>>
>