[petsc-dev] Kokkos/Crusher perforance

[Mark Adams]

On Sat, Jan 22, 2022 at 10:25 AM Jed Brown <jed at jedbrown.org> wrote:

> Mark Adams <mfadams at lbl.gov> writes:
>
> > On Fri, Jan 21, 2022 at 9:55 PM Barry Smith <bsmith at petsc.dev> wrote:
> >
> >>
> >> Interesting, Is this with all native Kokkos kernels or do some kokkos
> >> kernels use rocm?
> >>
> >
> > Ah, good question. I often run with tpl=0 but I did not specify here on
> > Crusher. In looking at the log files I see
> >
> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-crusher/externalpackages/git.kokkos-kernels/src/impl/tpls
> >
> > Here is a run with tpls turned off. These tpl includes are gone.
> >
> > It looks pretty much the same. A little slower but that could be noise.
>
> >
> ************************************************************************************************************************
> > ***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r
> -fCourier9' to print this document            ***
> >
> ************************************************************************************************************************
>
> We gotta say 160 chars because that's what we use now.
>
>
done

as far as streams, does it know to run on the GPU? You don't specify
something like -G 1 here for GPUs. I think you just get them all.


11:14 adams/aijkokkos-gpu-logging=
crusher:/gpfs/alpine/csc314/scratch/adams/petsc$ make
PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
PETSC_ARCH=arch-olcf-crusher streams
cc -o MPIVersion.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -fstack-protector -Qunused-arguments
-fvisibility=hidden -g -O3
 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
-I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-crusher/include
-I/opt/rocm-4.5.0/include    `pwd`/MPIVersion.c
Running streams with '/usr/bin/srun -p batch -N 1 -A csc314_crusher -t
00:10:00 ' using 'NPMAX=128'
1  53355.9207   Rate (MB/s)
2  39565.2208   Rate (MB/s) 0.741534
3  34538.3431   Rate (MB/s) 0.64732
4  32469.3375   Rate (MB/s) 0.608543
5  31041.1569   Rate (MB/s) 0.581776
6  30113.3826   Rate (MB/s) 0.564387
7  29562.5285   Rate (MB/s) 0.554063
8  29228.8090   Rate (MB/s) 0.547808
9  31474.3616   Rate (MB/s) 0.589895
10  31306.7647   Rate (MB/s) 0.586754
11  31147.4674   Rate (MB/s) 0.583768
12  31006.5008   Rate (MB/s) 0.581126
13  30859.4559   Rate (MB/s) 0.57837
14  30796.0587   Rate (MB/s) 0.577182
15  30604.4849   Rate (MB/s) 0.573591
16  30565.4340   Rate (MB/s) 0.572859
17  32421.9349   Rate (MB/s) 0.607654
18  34365.3424   Rate (MB/s) 0.644078
19  36289.4518   Rate (MB/s) 0.680139
20  38194.5300   Rate (MB/s) 0.715845
21  40160.4660   Rate (MB/s) 0.75269
22  42062.3931   Rate (MB/s) 0.788336
23  43890.2036   Rate (MB/s) 0.822593
24  45775.4680   Rate (MB/s) 0.857927
25  47708.8770   Rate (MB/s) 0.894163
26  49559.6810   Rate (MB/s) 0.928851
27  51457.5537   Rate (MB/s) 0.964421
28  53528.3420   Rate (MB/s) 1.00323
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220122/7d6aac9e/attachment-0001.html>