[petsc-users] Combining SUPERLU_DIST with CUDA fails
Xiaoye S. Li
xsli at lbl.gov
Tue Dec 21 12:35:20 CST 2021
Seems -isystem is a gcc option.
Can you try not to give "-I/usr/local/cuda-11.5/include" in
CUDA_ARCH_FLAGS, see whether nvcc can find the proper include/ directory?
Sherry
On Tue, Dec 21, 2021 at 10:26 AM Roland Richter <roland.richter at ntnu.no>
wrote:
> I'm already using GCC, together with Intel MPI, thus that should not be
> the problem. But I'll take a look if there are ways to fix that problem, or
> to circumvent it without having to install superlu_dist separately.
>
> Thanks!
>
> Regards,
>
> Roland
> Am 21.12.2021 um 19:04 schrieb Barry Smith:
>
>
> I think the problem comes from the following:
>
> -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61
> -DDEBUGlevel=0 -DPRNTlevel=0"
>
> which is ok, the location of the CUDA include files is passed in correctly
> but when the compile is done inside SuperLU_DIST's make
>
> cd
> ~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC
> && /usr/local/cuda-11.5/bin/nvcc -forward-unknown-to-host-compiler
> -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
> -I~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC
> -isystem=/usr/local/cuda-11.5/include -DUSE_VENDOR_BLAS
> -allow-unsupported-compiler -Xcompiler -fPIC -O3 -ccbin
> /opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17 -gencode
> arch=compute_61,code=sm_61 -Wno-deprecated-gpu-targets
> -I/opt/intel/oneapi/mpi/2021.5.0/include -O3 -DNDEBUG
> --generate-code=arch=compute_61,code=[compute_61,sm_61] -Xcompiler=-fPIC
> -std=c++11 -MD -MT SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
> -MF CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o.d -x cu -dc
> ~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/
> superlu_gpu_utils.cu -o CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>
> The
>
> -I/usr/local/cuda-11.5/include
>
> has been replaced with
>
> -isystem=/usr/local/cuda-11.5/include
>
> and I suspect the nvcc doesn't understand it so it ignores it. I assume
> somewhere inside the CMAKE processing it is replacing the -I with the
> -isystem=
>
> I don't know how one could fix this. I suggest you try installing
> SuperLU_DIST yourself directly. But I suspect the same problem will appear.
>
> It is possible that if you do not use the Intel compilers but use for
> example the GNU compilers the problem may not occur because cmake may not
> make the transformation.
>
> Barry
>
> Googling cmake isystem produces many messages about issues related to
> isystem and cmake but it would take a life-time to understand them.
>
>
> On Dec 21, 2021, at 2:24 AM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> I added/replaced all six lines which are in that commit, getting
>
> Executing: /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/opt/petsc
> -DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib"
> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> -DCMAKE_BUILD_TYPE=Release
> -DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
> -DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
> -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib
> -DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
> -DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
> -DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC
> -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native
> -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp"
> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC
> -std=gnu++17 -fopenmp"
> -DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
> -DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
> -DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp
> -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2
> -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch"
> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC
> -fopenmp -fallow-argument-mismatch" -DCMAKE_EXE_LINKER_FLAGS:STRING="
> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
> -DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64
> -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse
> -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda"
> -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61
> -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler -Xcompiler -fPIC -O3 -ccbin
> /opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17 -gencode
> arch=compute_61,code=sm_61 -Wno-deprecated-gpu-targets
> -I"/opt/intel/oneapi/mpi/2021.5.0/include" " -DUSE_XSDK_DEFAULTS=YES
> -DTPL_BLAS_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64
> -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse
> -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++
> -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread
> -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
> -L/opt/intel/oneapi/vpl/2022.0.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s
> -lquadmath" -DTPL_LAPACK_LIBRARIES="-lopenblas
> -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart
> -lcufft -lcublas -lcusparse -lcusolver -lcurand
> -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread
> -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
> -L/opt/intel/oneapi/vpl/2022.0.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s
> -lquadmath" -Denable_parmetislib=FALSE -DTPL_ENABLE_PARMETISLIB=FALSE
> -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0
> -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING=""
> -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
>
> but the compilation still fails with the same error.
> Am 20.12.21 um 22:46 schrieb Roland Richter:
>
> Yes, just checked, I only included the changes above the comment...
>
> Will test tomorrow, thanks for the help!
>
> Regards,
>
> Roland
> Am 20.12.2021 um 21:46 schrieb Barry Smith:
>
> Hmm, the fix should now be supplying a -DCMAKE_CUDA_FLAGS to this cmake
> command (line 48 at
> https://gitlab.com/petsc/petsc/-/merge_requests/4635/diffs) I do not see
> that flag being set inside the configure.log so I am guessing you didn't
> get the complete fix.
>
>
> Executing: /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/opt/petsc
> -DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib"
> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> -DCMAKE_BUILD_TYPE=Release
> -DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
> -DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
> -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib
> -DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp"
> -DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
> -DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
> -DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC
> -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native
> -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp"
> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC
> -std=gnu++17 -fopenmp"
> -DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
> -DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
> -DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp
> -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2
> -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch"
> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC
> -fopenmp -fallow-argument-mismatch" -DCMAKE_EXE_LINKER_FLAGS:STRING="
> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
> -DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64
> -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse
> -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda"
> -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61
> -DDEBUGlevel=0 -DPRNTlevel=0" -DUSE_XSDK_DEFAULTS=YES
> -DTPL_BLAS_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64
> -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse
> -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++
> -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread
> -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
> -L/opt/intel/oneapi/vpl/2022.0.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s
> -lquadmath" -DTPL_LAPACK_LIBRARIES="-lopenblas
> -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart
> -lcufft -lcublas -lcusparse -lcusolver -lcurand
> -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread
> -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
> -L/opt/intel/oneapi/vpl/2022.0.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s
> -lquadmath" -Denable_parmetislib=FALSE -DTPL_ENABLE_PARMETISLIB=FALSE
> -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0
> -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING=""
> -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
>
>
>
>
> On Dec 20, 2021, at 2:59 PM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> I introduced the changes from that patch directly, without checking out.
> Is that insufficient?
>
> Regards,
>
> Roland
> Am 20.12.2021 um 20:38 schrieb Barry Smith:
>
>
> Are you sure you have the correct PETSc branch? From configure.log it
> has
>
> Defined "VERSION_GIT" to ""v3.16.2-466-g959e1fce86""
> Defined "VERSION_DATE_GIT" to ""2021-12-18 11:17:24 -0600""
> Defined "VERSION_BRANCH_GIT" to ""master""
>
> It should have balay/slu-without-omp-3 for the branch.
>
>
>
> On Dec 20, 2021, at 10:50 AM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> In that case it fails with
>
> *~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/cublas_utils.h:22:10:
> fatal error: cublas_v2.h: No such file or directory*
>
> even though this header is available. I assume some header paths are not
> set correctly?
>
> Thanks,
>
> regards,
>
> Roland
> Am 20.12.21 um 16:29 schrieb Barry Smith:
>
>
> Please try the branch balay/slu-without-omp-3 It is in MR
> https://gitlab.com/petsc/petsc/-/merge_requests/4635
>
>
>
> On Dec 20, 2021, at 8:14 AM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> Hei,
>
> I tried to combine CUDA with superlu_dist in petsc using the following
> configure-line:
>
> *./configure PETSC_ARCH=mpich-complex-linux-gcc-demo
> --CC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc
> --CXX=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx
> --FC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc --CFLAGS="-mavx2
> -march=native -O3" --CXXFLAGS="-mavx2 -march=native -O3" --FFLAGS="-mavx2
> -march=native -O3" --CUDAFLAGS=-allow-unsupported-compiler --CUDA-CXX=g++
> --prefix=/opt/petsc --with-blaslapack=1 --with-mpi=1
> --with-scalar-type=complex --download-suitesparse=1 --with-cuda
> --with-debugging=0 --with-openmp --download-superlu_dist --force*
>
> but the configure-step fails with several errors correlated with CUDA and
> superlu_dist, the first one being
>
> *cublas_utils.c:21:37: error: ‘CUDART_VERSION’ undeclared (first use in
> this function); did you mean ‘CUDA_VERSION’?*
> * 21 | printf("CUDA version: v %d\n",CUDART_VERSION);*
> * | ^~~~~~~~~~~~~~*
> * | CUDA_VERSION*
>
> Compiling superlu_dist separately works, though (including CUDA).
>
> Is there a bug somewhere in the configure-routine? I attached the full
> configure-log.
>
> Thanks!
>
> Regards,
>
> Roland
> <configure.log>
>
>
> <configure.log>
>
>
>
> <configure.log>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211221/629d100d/attachment-0001.html>
More information about the petsc-users
mailing list