[petsc-dev] why does veccuda.py exist?

[Smith, Barry F.]

  Now that we are testing things, the GPU stuff has to actually work instead of just being a facade for funding agencies that think GPUs are worth spending money on ;-)


> On Feb 2, 2018, at 11:32 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Thanks. I'll try to be patient.
> 
> 
>> On Feb 2, 2018, at 11:28 PM, Karl Rupp <rupp at iue.tuwien.ac.at> wrote:
>> 
>> 
>>>>> Why can't the VECCUDA type coexist with the VECCUSP or VECVIENNACL types? If it can't coexist, can the code be reworked to allow it to coexist?
>>>> 
>>>> Currently it can't coexist because some variables are conditionally compiled and may be multiply defined (e.g. spptr).
>>>  Hmm, I don't think so. The use of spprt shouldn't mean there cannot be both VECCUDA and VECCUSP at the same time (with different vectors obviously).
>> 
>> There is no fundamental reason why it can't work. The relevant GPU code just hasn't been cleaned up yet.
>> 
>> Currently we have in vecimpl.h:
>> 
>> #if defined(PETSC_HAVE_CUSP)
>> PetscCUSPFlag          valid_GPU_array;    /* indicates where the most recently modified vector data is (GPU or CPU) */
>> void                   *spptr; /* if we're using CUSP, then this is the special pointer to the array on the GPU */
>> #elif defined(PETSC_HAVE_VIENNACL)
>> PetscViennaCLFlag      valid_GPU_array;    /* indicates where the most recently modified vector data is (GPU or CPU) */
>> void                   *spptr; /* if we're using ViennaCL, then this is the special pointer to the array on the GPU */
>> #elif defined(PETSC_HAVE_VECCUDA)
>> PetscCUDAFlag          valid_GPU_array;    /* indicates where the most recently modified vector data is (GPU or CPU) */
>> void                   *spptr; /* if we're using CUDA, then this is the special pointer to the array on the GPU */
>> #endif
>> 
>> 
>> Words can't tell how ugly this is. Fixing the spptr-thing is trivial, valid_GPU_array requires a bit more work to achieve consistent behavior across multiple different GPU vector types.
>> 
>> 
>>>>> I'm willing to do the refactorization and simplification but I need to know there is not some secret reason for these complications.
>>>> 
>>>> Unless you have to deliver something specific within the next few days, I'll (finally!) do it next week together with getting rid of VECCUSP.
>>>   So are we giving up on CUSP? And just using CUDA directly and ViennaCL?
>> 
>> We don't need both VECCUDA and VECCUSP. VECCUDA does not require an external library (part of the CUDA SDK!) and is at least as fast as VECCUSP, so the latter is obsolete (feature-wise they are the same).
>> 
>> However, I'm not saying that we give up on CUSP completely. CUSP's SA-AMG preconditioner is still useful. It just doesn't need a separate VECCUSP backend to operate (just like e.g. Hypre doesn't need a separate VECHYPRE).
>> 
>> 
>>>> These two steps should be done concurrently to avoid needless work.
>>>   There is no hurry; except I hate ugliness hanging around once I see it ;( It makes my skin itch, just knowing it exists ;)
>> 
>> Pain relief is on the way! ;-)
>> 
>> Best regards,
>> Karli
>