[petsc-dev] CUDA + OpenMP on Summit with Hypre

[Junchao Zhang]

On Sat, Nov 13, 2021 at 2:24 PM Mark Adams <mfadams at lbl.gov> wrote:

> I have a user that wants CUDA + Hypre on Sumit and they want to use OpenMP
> in their code. I configured with openmp but without thread safety and got
> this error.
>
> Maybe there is no need for us to do anything with omp in our
> configuration. Not sure.
>
> 15:08 main= summit:/gpfs/alpine/csc314/scratch/adams/petsc$ make
> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-opt-gcc9.1.0-omp-cuda11.0.3
> PETSC_ARCH="" check
> Running check examples to verify correct installation
> Using
> PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-opt-gcc9.1.0-omp-cuda11.0.3
> and PETSC_ARCH=
> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
> Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> [1] (280696) Warning: Could not find key lid0:0:2 in cache
> <=========================
> [1] (280696) Warning: Could not find key qpn0:0:0:2 in cache
> <=========================
> Unable to connect queue-pairs
> [h37n08:280696] Error: common_pami.c:1094 - ompi_common_pami_init() 1:
> Unable to create 1 PAMI communication context(s) rc=1
>
I don't know what petsc's thread safety is.  But this error seems to be in
the environment.   You can report to OLCF help.


> --------------------------------------------------------------------------
> No components were able to be opened in the pml framework.
>
> This typically means that either no components of this type were
> installed, or none of the installed components can be loaded.
> Sometimes this means that shared libraries required by these
> components are unable to be found/loaded.
>
>   Host:      h37n08
>   Framework: pml
> --------------------------------------------------------------------------
> [h37n08:280696] PML pami cannot be selected
> 1,5c1,16
> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> <   0 SNES Function norm 0.0406612
> <   1 SNES Function norm 4.12227e-06
> <   2 SNES Function norm 6.098e-11
> < Number of SNES iterations = 2
> ---
> > [1] (280721) Warning: Could not find key lid0:0:2 in cache
> <=========================
> > [1] (280721) Warning: Could not find key qpn0:0:0:2 in cache
> <=========================
> > Unable to connect queue-pairs
> > [h37n08:280721] Error: common_pami.c:1094 - ompi_common_pami_init() 1:
> Unable to create 1 PAMI communication context(s) rc=1
> >
> --------------------------------------------------------------------------
> > No components were able to be opened in the pml framework.
> >
> > This typically means that either no components of this type were
> > installed, or none of the installed components can be loaded.
> > Sometimes this means that shared libraries required by these
> > components are unable to be found/loaded.
> >
> >   Host:      h37n08
> >   Framework: pml
> >
> --------------------------------------------------------------------------
> > [h37n08:280721] PML pami cannot be selected
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
> Possible problem with ex19 running with hypre, diffs above
> =========================================
> 2,15c2,15
> <   0 SNES Function norm 2.391552133017e-01
> <     0 KSP Residual norm 2.325621076120e-01
> <     1 KSP Residual norm 1.654206318674e-02
> <     2 KSP Residual norm 7.202836119880e-04
> <     3 KSP Residual norm 1.796861424199e-05
> <     4 KSP Residual norm 2.461332992052e-07
> <   1 SNES Function norm 6.826585648929e-05
> <     0 KSP Residual norm 2.347339172985e-05
> <     1 KSP Residual norm 8.356798075993e-07
> <     2 KSP Residual norm 1.844045309619e-08
> <     3 KSP Residual norm 5.336386977405e-10
> <     4 KSP Residual norm 2.662608472862e-11
> <   2 SNES Function norm 6.549682264799e-11
> < Number of SNES iterations = 2
> ---
> > [0]PETSC ERROR: PETSc is configured with GPU support, but your MPI is
> not GPU-aware. For better performance, please use a GPU-aware MPI.
> > [0]PETSC ERROR: If you do not care, add option -use_gpu_aware_mpi 0. To
> not see the message again, add the option to your .petscrc, OR add it to
> the env var PETSC_OPTIONS.
> > [0]PETSC ERROR: If you do care, for IBM Spectrum MPI on OLCF Summit, you
> may need jsrun --smpiargs=-gpu.
> > [0]PETSC ERROR: For OpenMPI, you need to configure it --with-cuda (
> https://www.open-mpi.org/faq/?category=buildcuda)
> > [0]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 (
> http://mvapich.cse.ohio-state.edu/userguide/gdr/)
> > [0]PETSC ERROR: For Cray-MPICH, you need to set
> MPICH_RDMA_ENABLED_CUDA=1 (
> https://www.olcf.ornl.gov/tutorials/gpudirect-mpich-enabled-cuda/)
> >
> --------------------------------------------------------------------------
> > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
> > with errorcode 76.
> >
> > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> > You may or may not see output from other processes, depending on
> > exactly when Open MPI kills them.
> >
> --------------------------------------------------------------------------
> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
> Possible problem with ex19 running with cuda, diffs above
> =========================================
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20211113/a3547522/attachment.html>