[petsc-dev] Kokkos/Crusher perforance

Sun Jan 23 22:44:22 CST 2022

  Junchao,

     Without GPU aware MPI, is it moving the entire vector to the CPU and doing the scatter and moving everything back or does it just move up exactly what needs to be sent to the other ranks and move back exactly what it received from other ranks?

    It is moving 4.74e+02 * 1e+6 bytes total data up and then down. Is that a reasonable amount?

    Why is it moving 800 distinct counts up and 800 distinct counts down when the MatMult is done 400 times, shouldn't it be 400 counts?

  Mark,

     Can you run both with GPU aware MPI?

  Norm, AXPY, pointwisemult roughly the same.

> On Jan 23, 2022, at 11:24 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> Ugh, try again. Still a big difference, but less.  Mat-vec does not change much.
> 
> On Sun, Jan 23, 2022 at 7:12 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
> 
>  You have debugging turned on on crusher but not permutter
> 
>> On Jan 23, 2022, at 6:37 PM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> * Perlmutter is roughly 5x faster than Crusher on the one node 2M eq test. (small)
>> This is with 8 processes. 
>> 
>> * The next largest version of this test, 16M eq total and 8 processes, fails in memory allocation in the mat-mult setup in the Kokkos Mat.
>> 
>> * If I try to run with 64 processes on Perlmutter I get this error in initialization. These nodes have 160 Gb of memory.
>> (I assume this is related to these large memory requirements from loading packages, etc....)
>> 
>> Thanks,
>> Mark
>> 
>> + srun -n64 -N1 --cpu-bind=cores --ntasks-per-core=1 ../ex13 -dm_plex_box_faces 4,4,4 -petscpartitioner_simple_process_grid 4,4,4 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_refine 6 -dm_view -pc_type jacobi -log
>> _view -ksp_view -use_gpu_aware_mpi false -dm_mat_type aijkokkos -dm_vec_type kokkos -log_trace
>> + tee jac_out_001_kokkos_Perlmutter_6_8.txt
>> [48]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [48]PETSC ERROR: GPU error 
>> [48]PETSC ERROR: cuda error 2 (cudaErrorMemoryAllocation) : out of memory
>> [48]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting.
>> [48]PETSC ERROR: Petsc Development GIT revision: v3.16.3-683-gbc458ed4d8  GIT Date: 2022-01-22 12:18:02 -0600
>> [48]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tests/data/../ex13 on a arch-perlmutter-opt-gcc-kokkos-cuda named nid001424 by madams Sun Jan 23 15:19:56 2022
>> [48]PETSC ERROR: Configure options --CFLAGS="   -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLAN
>> DAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --with-cc=cc --with-cxx=CC --with-fc=ftn --LDFLAGS=-lmpifort_gnu_91 --with-cudac=/global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc --COPTFLAGS="   -O3" --CXXOPTFLAGS=" -O3" --FOPTFLAGS="   -O3"
>>  --with-debugging=0 --download-metis --download-parmetis --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --download-kokkos --download-kokkos-kernels --with-kokkos-kernels-tpl=0 --with-
>> make-np=8 PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
>> [48]PETSC ERROR: #1 initialize() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:72
>> [48]PETSC ERROR: #2 initialize() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:343
>> [48]PETSC ERROR: #3 PetscDeviceInitializeTypeFromOptions_Private() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:319
>> [48]PETSC ERROR: #4 PetscDeviceInitializeFromOptions_Internal() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:449
>> [48]PETSC ERROR: #5 PetscInitialize_Common() at /global/u2/m/madams/petsc/src/sys/objects/pinit.c:963
>> [48]PETSC ERROR: #6 PetscInitialize() at /global/u2/m/madams/petsc/src/sys/objects/pinit.c:1238
>> 
>> 
>> On Sun, Jan 23, 2022 at 8:58 AM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> 
>> On Sat, Jan 22, 2022 at 6:22 PM Barry Smith <bsmith at petsc.dev <mailto:bsmith at petsc.dev>> wrote:
>> 
>>    I cleaned up Mark's last run and put it in a fixed-width font. I realize this may be too difficult but it would be great to have identical runs to compare with on Summit.
>> 
>> I was planning on running this on Perlmutter today, as well as some sanity checks like all GPUs are being used. I'll try PetscDeviceView.
>> 
>> Junchao modified the timers and all GPU > CPU now, but he seemed to move the timers more outside and Barry wants them tight on the "kernel".
>> I think Junchao is going to work on that so I will hold off.
>> (I removed the the Kokkos wait stuff and seemed to run a little faster but I am not sure how deterministic the timers are, and I did a test with GAMG and it was fine.)
>> 
>> 
>> 
>>    As Jed noted Scatter takes a long time but the pack and unpack take no time? Is this not timed if using Kokkos?
>> 
>> 
>> --- Event Stage 2: KSP Solve only
>> 
>> MatMult              400 1.0 8.8003e+00 1.1 1.06e+11 1.0 2.2e+04 8.5e+04 0.0e+00  2 55 61 54  0  70 91100100   95,058   132,242      0 0.00e+00    0 0.00e+00 100
>> VecScatterBegin      400 1.0 1.3391e+00 2.6 0.00e+00 0.0 2.2e+04 8.5e+04 0.0e+00  0  0 61 54  0   7  0100100        0         0      0 0.00e+00    0 0.00e+00  0
>> VecScatterEnd        400 1.0 1.3240e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   9  0  0  0        0         0      0 0.00e+00    0 0.00e+00  0
>> SFPack               400 1.0 1.8276e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0        0         0      0 0.00e+00    0 0.00e+00  0
>> SFUnpack             400 1.0 6.2653e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0        0         0      0 0.00e+00    0 0.00e+00  0
>> 
>> KSPSolve               2 1.0 1.2540e+01 1.0 1.17e+11 1.0 2.2e+04 8.5e+04 1.2e+03  3 60 61 54 60 100100100      73,592   116,796      0 0.00e+00    0 0.00e+00 100
>> VecTDot              802 1.0 1.3551e+00 1.2 3.36e+09 1.0 0.0e+00 0.0e+00 8.0e+02  0  2  0  0 40  10  3  0      19,627    52,599      0 0.00e+00    0 0.00e+00 100
>> VecNorm              402 1.0 9.0151e-01 2.2 1.69e+09 1.0 0.0e+00 0.0e+00 4.0e+02  0  1  0  0 20   5  1  0  0   14,788   125,477      0 0.00e+00    0 0.00e+00 100
>> VecAXPY              800 1.0 8.2617e-01 1.0 3.36e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   7  3  0  0   32,112    61,644      0 0.00e+00    0 0.00e+00 100
>> VecAYPX              398 1.0 8.1525e-01 1.6 1.67e+09 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   5  1  0  0   16,190    20,689      0 0.00e+00    0 0.00e+00 100
>> VecPointwiseMult     402 1.0 3.5694e-01 1.0 8.43e+08 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   3  1  0  0   18,675    38,633      0 0.00e+00    0 0.00e+00 100
>> 
>> 
>> 
>>> On Jan 22, 2022, at 12:40 PM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>>> 
>>> And I have a new MR with if you want to see what I've done so far.
>> 
>> <jac_out_001_kokkos_Crusher_6_1_notpl.txt><jac_out_001_kokkos_Perlmutter_6_1.txt>
> 
> <jac_out_001_kokkos_Crusher_6_1_notpl.txt>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220123/e3df1260/attachment.html>