[petsc-users] Combining SUPERLU_DIST with CUDA fails

Barry Smith bsmith at petsc.dev
Tue Dec 21 13:21:11 CST 2021 

  Roland,

    The quickest way for you to debug is to go into $PETSC_DIR/$PETSC_ARCH/externalpackages/git.superlu_dist/petsc-build and then cut and paste the CMAKE command from the configure.log file (that we pass to SuperLU_DIST). You can then edit the command to try different options (as Sherry suggests first try removing the -I business). After running the cmake command you can then cut and paste the make command from configure.log

   This interactive approach is much faster for debugging cmake problems then trying to modify PETSc to get a proper cmake command for superlu_dist. Once you figure out the magic juice you can tell us and we can figure out how to add it to PETSc's process for building SuperLU_DIST.

  Barry


> On Dec 21, 2021, at 1:35 PM, Xiaoye S. Li <xsli at lbl.gov> wrote:
> 
> Seems -isystem is a gcc option. 
> Can you try not to give "-I/usr/local/cuda-11.5/include" in CUDA_ARCH_FLAGS, see whether nvcc can find the proper include/ directory?
> 
> Sherry
> 
> On Tue, Dec 21, 2021 at 10:26 AM Roland Richter <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
> I'm already using GCC, together with Intel MPI, thus that should not be the problem. But I'll take a look if there are ways to fix that problem, or to circumvent it without having to install superlu_dist separately.
> 
> Thanks!
> 
> Regards,
> 
> Roland
> 
> Am 21.12.2021 um 19:04 schrieb Barry Smith:
>> 
>>   I think the problem comes from the following: 
>> 
>>> -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0"
>>> 
>> which is ok, the location of the CUDA include files is passed in correctly but when the compile is done inside SuperLU_DIST's make
>> 
>> cd ~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC && /usr/local/cuda-11.5/bin/nvcc -forward-unknown-to-host-compiler -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS -I~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC -isystem=/usr/local/cuda-11.5/include -DUSE_VENDOR_BLAS -allow-unsupported-compiler -Xcompiler -fPIC -O3 -ccbin /opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17 -gencode arch=compute_61,code=sm_61  -Wno-deprecated-gpu-targets -I/opt/intel/oneapi/mpi/2021.5.0/include -O3 -DNDEBUG --generate-code=arch=compute_61,code=[compute_61,sm_61] -Xcompiler=-fPIC -std=c++11 -MD -MT SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o -MF CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o.d -x cu -dc ~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> -o CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>> 
>> The 
>> 
>>  -I/usr/local/cuda-11.5/include
>> 
>> has been replaced with 
>> 
>> -isystem=/usr/local/cuda-11.5/include 
>> 
>> and I suspect the nvcc doesn't understand it so it ignores it. I assume somewhere inside the CMAKE processing it is replacing the -I with the -isystem=
>> 
>> I don't know how one could fix this.  I suggest you try installing SuperLU_DIST yourself directly. But I suspect the same problem will appear.
>> 
>> It is possible that if you do not use the Intel compilers but use for example the GNU compilers the problem may not occur because cmake may not make the transformation.
>> 
>> Barry
>> 
>> Googling cmake isystem produces many messages about issues related to isystem and cmake but it would take a life-time to understand them.
>> 
>> 
>>> On Dec 21, 2021, at 2:24 AM, Roland Richter <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>> 
>>> I added/replaced all six lines which are in that commit, getting 
>>> 
>>> Executing: /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/opt/petsc -DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib" -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc" -DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx" -DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc" -DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc" -DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE -DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda" -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0" -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler -Xcompiler -fPIC -O3 -ccbin /opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17 -gencode arch=compute_61,code=sm_61  -Wno-deprecated-gpu-targets -I"/opt/intel/oneapi/mpi/2021.5.0/include" " -DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -L/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11 -L/usr/lib64/gcc/x86_64-suse-linux/11 -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib -L/opt/intel/oneapi/vpl/2022.0.0/lib -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s -lquadmath" -DTPL_LAPACK_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -L/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11 -L/usr/lib64/gcc/x86_64-suse-linux/11 -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib -L/opt/intel/oneapi/vpl/2022.0.0/lib -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s -lquadmath" -Denable_parmetislib=FALSE -DTPL_ENABLE_PARMETISLIB=FALSE -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
>>> 
>>> but the compilation still fails with the same error.
>>> 
>>> Am 20.12.21 um 22:46 schrieb Roland Richter:
>>>> Yes, just checked, I only included the changes above the comment...
>>>> 
>>>> Will test tomorrow, thanks for the help!
>>>> 
>>>> Regards,
>>>> 
>>>> Roland
>>>> 
>>>> Am 20.12.2021 um 21:46 schrieb Barry Smith:
>>>>>   Hmm, the fix should now be supplying a -DCMAKE_CUDA_FLAGS to this cmake command (line 48 at https://gitlab.com/petsc/petsc/-/merge_requests/4635/diffs <https://gitlab.com/petsc/petsc/-/merge_requests/4635/diffs>) I do not see that flag being set inside the configure.log so I am guessing you didn't get the complete fix.
>>>>> 
>>>>> 
>>>>> Executing: /usr/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/opt/petsc -DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib" -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc" -DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp" -DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx" -DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fopenmp -fPIC -std=gnu++17 -fopenmp" -DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc" -DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc" -DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3 -fPIC -fopenmp -fallow-argument-mismatch" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE -DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda" -DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include -arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0" -DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -L/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11 -L/usr/lib64/gcc/x86_64-suse-linux/11 -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib -L/opt/intel/oneapi/vpl/2022.0.0/lib -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s -lquadmath" -DTPL_LAPACK_LIBRARIES="-lopenblas -Wl,-rpath,/usr/local/cuda-11.5/lib64 -L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++ -ldl -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -L/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11 -L/usr/lib64/gcc/x86_64-suse-linux/11 -Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib -L/opt/intel/oneapi/vpl/2022.0.0/lib -Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8 -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib -Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -L/opt/intel/oneapi/dal/2021.5.1/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib -L/opt/intel/oneapi/compiler/2022.0.1/linux/lib -Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -L/opt/intel/oneapi/clck/2021.5.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release -Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran -lm -lgcc_s -lquadmath" -Denable_parmetislib=FALSE -DTPL_ENABLE_PARMETISLIB=FALSE -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Dec 20, 2021, at 2:59 PM, Roland Richter <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>>>>> 
>>>>>> I introduced the changes from that patch directly, without checking out. Is that insufficient?
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Roland
>>>>>> 
>>>>>> Am 20.12.2021 um 20:38 schrieb Barry Smith:
>>>>>>> 
>>>>>>>   Are you sure you have the correct PETSc branch? From configure.log it has
>>>>>>> 
>>>>>>>             Defined "VERSION_GIT" to ""v3.16.2-466-g959e1fce86""
>>>>>>>             Defined "VERSION_DATE_GIT" to ""2021-12-18 11:17:24 -0600""
>>>>>>>             Defined "VERSION_BRANCH_GIT" to ""master""
>>>>>>> 
>>>>>>> It should have balay/slu-without-omp-3 for the branch.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Dec 20, 2021, at 10:50 AM, Roland Richter <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>>>>>>> 
>>>>>>>> In that case it fails with
>>>>>>>> 
>>>>>>>> ~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/cublas_utils.h:22:10: fatal error: cublas_v2.h: No such file or directory
>>>>>>>> 
>>>>>>>> even though this header is available. I assume some header paths are not set correctly?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> regards,
>>>>>>>> 
>>>>>>>> Roland
>>>>>>>> 
>>>>>>>> Am 20.12.21 um 16:29 schrieb Barry Smith:
>>>>>>>>> 
>>>>>>>>>   Please try the branch balay/slu-without-omp-3  It is in MR https://gitlab.com/petsc/petsc/-/merge_requests/4635 <https://gitlab.com/petsc/petsc/-/merge_requests/4635>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Dec 20, 2021, at 8:14 AM, Roland Richter <roland.richter at ntnu.no <mailto:roland.richter at ntnu.no>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hei,
>>>>>>>>>> 
>>>>>>>>>> I tried to combine CUDA with superlu_dist in petsc using the following configure-line:
>>>>>>>>>> 
>>>>>>>>>> ./configure PETSC_ARCH=mpich-complex-linux-gcc-demo --CC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc --CXX=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx --FC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc --CFLAGS="-mavx2 -march=native -O3" --CXXFLAGS="-mavx2 -march=native -O3" --FFLAGS="-mavx2 -march=native -O3" --CUDAFLAGS=-allow-unsupported-compiler --CUDA-CXX=g++ --prefix=/opt/petsc --with-blaslapack=1 --with-mpi=1 --with-scalar-type=complex --download-suitesparse=1 --with-cuda --with-debugging=0 --with-openmp --download-superlu_dist --force
>>>>>>>>>> 
>>>>>>>>>> but the configure-step fails with several errors correlated with CUDA and superlu_dist, the first one being
>>>>>>>>>> 
>>>>>>>>>> cublas_utils.c:21:37: error: ‘CUDART_VERSION’ undeclared (first use in this function); did you mean ‘CUDA_VERSION’?
>>>>>>>>>>    21 |     printf("CUDA version:   v %d\n",CUDART_VERSION);
>>>>>>>>>>       |                                     ^~~~~~~~~~~~~~
>>>>>>>>>>       |                                     CUDA_VERSION
>>>>>>>>>> 
>>>>>>>>>> Compiling superlu_dist separately works, though (including CUDA).
>>>>>>>>>> 
>>>>>>>>>> Is there a bug somewhere in the configure-routine? I attached the full configure-log.
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> 
>>>>>>>>>> Roland
>>>>>>>>>> 
>>>>>>>>>> <configure.log>
>>>>>>>>> 
>>>>>>>> <configure.log>
>>>>>>> 
>>>>> 
>>> <configure.log>
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211221/f8ed394d/attachment-0001.html>

More information about the petsc-users mailing list