[petsc-dev] sm_70

[Mark Adams]

That was a login node. Here is a compute node on SUMMIT:

12:45 130 adams/feature-mat-cuda *= ~/petsc/src/mat/tutorials$ nvidia-smi
Sat Sep 26 12:45:28 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.116.00   Driver Version: 418.116.00   CUDA Version: 10.1
  |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.
ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute
M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000004:04:00.0 Off |
 0 |
| N/A   36C    P0    36W / 300W |      0MiB / 16130MiB |      0%   E.
Process |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000004:05:00.0 Off |
 0 |
| N/A   41C    P0    36W / 300W |      0MiB / 16130MiB |      0%   E.
Process |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000035:03:00.0 Off |
 0 |
| N/A   34C    P0    36W / 300W |      0MiB / 16130MiB |      0%   E.
Process |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000035:04:00.0 Off |
 0 |
| N/A   42C    P0    37W / 300W |      0MiB / 16130MiB |      0%   E.
Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU
Memory |
|  GPU       PID   Type   Process name                             Usage
   |
|=============================================================================|
|  No running processes found
  |
+-----------------------------------------------------------------------------+

On Sat, Sep 26, 2020 at 12:44 PM Mark Adams <mfadams at lbl.gov> wrote:

> 0:30 adams/feature-mat-cuda *= ~/petsc$ nvidia-smi
> Sat Sep 26 12:42:37 2020
>
> +-----------------------------------------------------------------------------+
> | NVIDIA-SMI 418.116.00   Driver Version: 418.116.00   CUDA Version: 10.1
>     |
>
> |-------------------------------+----------------------+----------------------+
> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.
> ECC |
> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute
> M. |
>
> |===============================+======================+======================|
> |   0  Tesla V100-SXM2...  On   | 00000004:05:00.0 Off |
>  0 |
> | N/A   38C    P0    36W / 300W |      0MiB / 16130MiB |      0%   E.
> Process |
>
> +-------------------------------+----------------------+----------------------+
> |   1  Tesla V100-SXM2...  On   | 00000035:04:00.0 Off |
>  0 |
> | N/A   40C    P0    38W / 300W |      0MiB / 16130MiB |      0%   E.
> Process |
>
> +-------------------------------+----------------------+----------------------+
>
>
> +-----------------------------------------------------------------------------+
> | Processes:                                                       GPU
> Memory |
> |  GPU       PID   Type   Process name                             Usage
>    |
>
> |=============================================================================|
> |  No running processes found
>     |
>
> +-----------------------------------------------------------------------------+
>
> On Fri, Sep 25, 2020 at 9:36 PM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>
>>  Configure by default should find out the available GPU and build for
>> that sm_*  it should not require the user to set this (how the heck is the
>> user going to know what to set?)  If I remember correctly there is a
>> utility available that gives this information.
>>
>> For CUDA I believe the tool is nvidia-smi. Should make sure this
>> automatic detection works when configuring —with-batch though since login
>> nodes might have different arch than compute.
>>
>> Best regards,
>>
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>> Cell: (312) 694-3391
>>
>> On Sep 25, 2020, at 21:09, Barry Smith <bsmith at petsc.dev> wrote:
>>
>>
>>   Configure by default should find out the available GPU and build for
>> that sm_*  it should not require the user to set this (how the heck is the
>> user going to know what to set?)  If I remember correctly there is a
>> utility available that gives this information.
>>
>>   For generic builds like in package distributions I don't know how it
>> should work, ideally all the possibilities would be available in the
>> library and at run time the correct one will be utilized.
>>
>>   Barry
>>
>>
>> On Sep 25, 2020, at 5:49 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>    '--CUDAFLAGS=-arch=sm_70',
>>
>> seems to fix this.
>>
>> On Fri, Sep 25, 2020 at 6:31 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> I see kokkos and hyper have a sm_70 flag, but I don't see one for PETSc.
>>>
>>> It looks like you have to specify this to get modern atomics to work in
>>> Cuda. I get:
>>>
>>> /ccs/home/adams/petsc/include/petscaijdevice.h(99): error: no instance
>>> of overloaded function "atomicAdd" matches the argument list
>>>             argument types are: (double *, double)
>>>
>>> I tried using a Kokkos configuration, thinking I could get these sm_70
>>> flags, but that did not work.
>>>
>>> Any ideas?
>>>
>>> Mark
>>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20200926/5f7a13e1/attachment.html>