[petsc-users] Kokkos with OpenMP

[Barry Smith]

  Mark,

> On Jan 17, 2021, at 3:08 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> I am getting this Kokkos build failure with OpenMP.

Here is the problem. You set  --CXXOPTFLAGS="-g -O0 -fPIC  -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS  ./configure never determines the flag to put into CXXPPFLAGS.

It is more desirable if you do not provide -fPIC  -fopenmp.  ./configure will figure them out for you.

But yes it should still work. I don't know how to fix configure to handle this case cleanly. We could not use the CXXPPFLAGS but always use the compiler flags for checking includes, this would change a 20 year old decision, since CXXPPFLAGS is for the preprocess :-(.

Please remove the -fPIC  -fopenmp from your ./configure options and see if it succeeds. 

(see below for the answer to your second question).
----- 

TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86)
  Checks if a particular include file can be found along particular include paths
Checking for header files ['KokkosBlas.hpp', 'KokkosSparse_CrsMatrix.hpp'] in ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include']
                Checking include with compiler flags var CXXPPFLAGS ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include', '/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include']
Preprocessing source:
#include "confdefs.h"
#include "conffix.h"
#include <KokkosBlas.hpp>
Executing: mpicxx -E  -I/tmp/petsc-jEwQ8b/config.setCompilers -I/tmp/petsc-jEwQ8b/config.packages.kokkos -I/tmp/petsc-jEwQ8b/config.headers   -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include /tmp/petsc-jEwQ8b/config.headers/conftest.cc 
Possible ERROR while running preprocessor: exit code 1
stdout:
----


> Also, can you build Kokkos with OMP and Cuda? I know you can't have two execution spaces but can you use Kokkos-Cuda with OMP in the same executable?

  In theory you can do this but Barry enforced a packages design decision (for the sake of less configure options;) that does not allow it by default. Edit 
kokkos.py and remove the lines.

 if self.openmp.found + pthreadfound + self.cuda.found > 1:
      raise RuntimeError("Kokkos only supports a single parallel system during its configuration")

  if self.openmp.found:
      args.append('-DKokkos_ENABLE_OPENMP=ON')
      self.system = 'OpenMP'

  I guess we will need to add an option like --with-kokkos-device=CUDA or OpenMP or  prethread or HIP to allow the user to set a particular device when several are possible. Stefano pushed for this long ago but I resisted. 

  Barry


> Thanks,
> Mark
> <configure.log>