[petsc-dev] CUDA + OpenMP on Summit with Hypre

[Mark Adams]

I have a user that wants CUDA + Hypre on Sumit and they want to use OpenMP
in their code. I configured with openmp but without thread safety and got
this error.

Maybe there is no need for us to do anything with omp in our configuration.
Not sure.

15:08 main= summit:/gpfs/alpine/csc314/scratch/adams/petsc$ make
PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-opt-gcc9.1.0-omp-cuda11.0.3
PETSC_ARCH="" check
Running check examples to verify correct installation
Using
PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-opt-gcc9.1.0-omp-cuda11.0.3
and PETSC_ARCH=
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
[1] (280696) Warning: Could not find key lid0:0:2 in cache
<=========================
[1] (280696) Warning: Could not find key qpn0:0:0:2 in cache
<=========================
Unable to connect queue-pairs
[h37n08:280696] Error: common_pami.c:1094 - ompi_common_pami_init() 1:
Unable to create 1 PAMI communication context(s) rc=1
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      h37n08
  Framework: pml
--------------------------------------------------------------------------
[h37n08:280696] PML pami cannot be selected
1,5c1,16
< lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
<   0 SNES Function norm 0.0406612
<   1 SNES Function norm 4.12227e-06
<   2 SNES Function norm 6.098e-11
< Number of SNES iterations = 2
---
> [1] (280721) Warning: Could not find key lid0:0:2 in cache
<=========================
> [1] (280721) Warning: Could not find key qpn0:0:0:2 in cache
<=========================
> Unable to connect queue-pairs
> [h37n08:280721] Error: common_pami.c:1094 - ompi_common_pami_init() 1:
Unable to create 1 PAMI communication context(s) rc=1
> --------------------------------------------------------------------------
> No components were able to be opened in the pml framework.
>
> This typically means that either no components of this type were
> installed, or none of the installed components can be loaded.
> Sometimes this means that shared libraries required by these
> components are unable to be found/loaded.
>
>   Host:      h37n08
>   Framework: pml
> --------------------------------------------------------------------------
> [h37n08:280721] PML pami cannot be selected
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
Possible problem with ex19 running with hypre, diffs above
=========================================
2,15c2,15
<   0 SNES Function norm 2.391552133017e-01
<     0 KSP Residual norm 2.325621076120e-01
<     1 KSP Residual norm 1.654206318674e-02
<     2 KSP Residual norm 7.202836119880e-04
<     3 KSP Residual norm 1.796861424199e-05
<     4 KSP Residual norm 2.461332992052e-07
<   1 SNES Function norm 6.826585648929e-05
<     0 KSP Residual norm 2.347339172985e-05
<     1 KSP Residual norm 8.356798075993e-07
<     2 KSP Residual norm 1.844045309619e-08
<     3 KSP Residual norm 5.336386977405e-10
<     4 KSP Residual norm 2.662608472862e-11
<   2 SNES Function norm 6.549682264799e-11
< Number of SNES iterations = 2
---
> [0]PETSC ERROR: PETSc is configured with GPU support, but your MPI is not
GPU-aware. For better performance, please use a GPU-aware MPI.
> [0]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To
not see the message again, add the option to your .petscrc, OR add it to
the env var PETSC_OPTIONS.
> [0]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you
may need jsrun --smpiargs=-gpu.
> [0]PETSC ERROR: For OpenMPI, you need to configure it --with-cuda (
https://www.open-mpi.org/faq/?category=buildcuda)
> [0]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 (
http://mvapich.cse.ohio-state.edu/userguide/gdr/)
> [0]PETSC ERROR: For Cray-MPICH, you need to set MPICH_RDMA_ENABLED_CUDA=1
(https://www.olcf.ornl.gov/tutorials/gpudirect-mpich-enabled-cuda/)
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
> with errorcode 76.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
Possible problem with ex19 running with cuda, diffs above
=========================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20211113/73dde8c4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 16891 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20211113/73dde8c4/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2188212 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20211113/73dde8c4/attachment-0003.obj>