[petsc-dev] [petsc-maint] Installing with CUDA on a cluster

[Karl Rupp]

Hi Satish,

thanks for the pull request. I approve the changes, improved appending 
the -Wno-deprecated-gpu-targets to also work on my machine, and have 
merged everything to next.


> * My fixes should alleviate some of the CUSP installation issues. I
>    don't know enough about CUSP interface wrt the useful features vs
>    other burderns - and if its good to drop it or not. [If needed - we
>    can add in more version dependencies in configure]

This should be fine for now. In the long term CUSP may be completely 
superseded by NVIDIA's AMGX. Let's see how things develop...



> * Wrt CUDA - currently my test is with CUDA-7.5. I can try migrating a
>    couple of tests to CUDA-9.1 [on frog]. But what about older
>    releases?  Any reason we should drop them? I.e any reason to up the
>    following values?
> 
>      self.CUDAMinVersion   = '5000' # Minimal cuda version is 5.0
>      self.CUSPMinVersion  = '400' # Minimal cusp version is 0.4

See the answer here for a list of CUDA capabilities and defaults:
https://stackoverflow.com/questions/28932864/cuda-compute-capability-requirements

We definitely don't need to support compute architecture 1.x (~10 years 
old), as there is no double precision support and hence fairly useless 
for our purposes. Thus, we should be absolutely fine with requiring CUDA 
7.0 or higher.


> 
>    We do change it for complex build [we don't have a test for this case]
> 
>      if self.defaultScalarType.lower() == 'complex': self.CUDAMinVersion = '7050'

I don't remember the exact reason, but I remember that there is one for 
requiring CUDA 7.5 here. Let's use CUDA 7.5 as the minimum for both real 
and complex then?



> * Our test GUP is M2090 - with Compute capability (version) 2.0.
>    CUDA-7.5 works on it. CUDA-8 gives deprecated warnings. CUDA-9 does
>    not work? So what do we do for such old hardware? Do we keep
>    CUDA-7.5 is the minimum supported version for extended time? [At
>    some point we could switch to minimum version CUDA-8 - if we can get
>    rid of the warnings]

Your PR silences the deprecation warnings.
Compute capability 2.0 is fine for our tests for some time to come. We 
should certainly upgrade at some point, yet my experience with GPUs is 
that older GPUs are actually the better test environment, as they tend 
to reveal bugs quicker than newer hardware.

Best regards,
Karli


> 
> * BTW: Wrt --with-cuda-arch, I'm hoping we can get rid of it in favor
>    of CUDAFLAGS [with defaults similar to CFLAGS defaults] - but its
>    not clear if I can easily untangle the dependencies we have [wrt CPP
>    - and others]
>    
>    Or can we get rid of this default alltogether [currently
>    -arch=sm_20] - and expect nvcc to have sane defaults? Then we can
>    probably eliminate all this complicated code. [If cuda-7.5 and higer
>    do this properly - we could use that as the minimum supported version?]
> 
> Satish
> 
> On Sat, 10 Mar 2018, Karl Rupp wrote:
> 
>> Hi all,
>>
>> a couple of notes here, particularly for Manuel:
>>
>>   * CUSP is repeatedly causing such installation problems, hence we will soon
>> drop it as a vector backend and instead only provide a native
>> CUBLAS/CUSPARSE-based backend.
>>
>>   * you can use this native CUDA backend already now. Just configure with only
>> --with-cuda=1 --with-cuda-arch=sm_60 (sm_30 should also work and is compatible
>> with Tesla K20 GPUs you may find on other clusters).
>>
>>   * The multigrid preconditioner from CUSP is selected via
>>     -pc_type sa_cusp
>>     Make sure you also use -vec_type cusp -mat_type aijcusp
>>     If you don't need the multigrid preconditioner from CUSP, please
>> just reconfigure and use the native CUDA-backend with -vec_type cuda -mat_type
>> aijcusparse
>>
>>   * Right now only one of {native CUDA, CUSP, ViennaCL} can be activated at
>> configure time. This will be fixed later this month.
>>
>> If you're looking for a GPU-accelerated multigrid preconditioner: I just heard
>> yesterday that NVIDIA's AMGX is now open source. I'll provide a wrapper within
>> PETSc soon.
>>
>> As Matt already said: Don't expect much more than a modest speedup over your
>> existing CPU-based code - provided that your setup is GPU-friendly and your
>> problem size is appropriate.
>>
>> Best regards,
>> Karli
>>
>>
>>
>>
>> On 03/10/2018 03:38 AM, Satish Balay wrote:
>>> I've updated configure so that --download-cusp gets the
>>> correct/compatible cusp version - for cuda 7,8 vs 9
>>>
>>> The changes are in branch balay/cuda-cusp-cleanup - and merged to next.
>>>
>>> Satish
>>>
>>> On Wed, 7 Mar 2018, Satish Balay wrote:
>>>
>>>> --download-cusp gets hardly ever used so likely broken.
>>>>
>>>> It needs to be updated to somehow use the correct cusp version based
>>>> on the cuda version thats being used.
>>>>
>>>> [and since we can't easily check for cusp compatibility - we should
>>>> probably
>>>> remove checkCUSPVersion() code]
>>>>
>>>> When using Cuda-9 - you can try options:
>>>>
>>>> --download-cusp=1 --download-cusp-commit=116b090
>>>>
>>
>>
>