<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Roland,<div class=""><br class=""></div><div class=""> The quickest way for you to debug is to go into $PETSC_DIR/$PETSC_ARCH/externalpackages/git.superlu_dist/petsc-build and then cut and paste the CMAKE command from the configure.log file (that we pass to SuperLU_DIST). You can then edit the command to try different options (as Sherry suggests first try removing the -I business). After running the cmake command you can then cut and paste the make command from configure.log</div><div class=""><br class=""></div><div class=""> This interactive approach is much faster for debugging cmake problems then trying to modify PETSc to get a proper cmake command for superlu_dist. Once you figure out the magic juice you can tell us and we can figure out how to add it to PETSc's process for building SuperLU_DIST.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 21, 2021, at 1:35 PM, Xiaoye S. Li <<a href="mailto:xsli@lbl.gov" class="">xsli@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_default" style="font-family: verdana, sans-serif; font-size: small;">Seems <span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class="">-isystem is a gcc option. </span></div><div class="gmail_default" style="font-family: verdana, sans-serif; font-size: small;"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class="">Can you try not to give "</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class="">-I/usr/</span><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class="">local/cuda-11.5/include" in CUDA_ARCH_FLAGS, see whether nvcc can find the proper include/ directory?</span></div><div class="gmail_default" style="font-family: verdana, sans-serif; font-size: small;"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class=""><br class=""></span></div><div class="gmail_default" style="font-family: verdana, sans-serif; font-size: small;"><span style="font-family:Arial,Helvetica,sans-serif;color:rgb(34,34,34)" class="">Sherry</span></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 21, 2021 at 10:26 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no" class="">roland.richter@ntnu.no</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class=""><p class="">I'm already using GCC, together with Intel MPI, thus that should
not be the problem. But I'll take a look if there are ways to fix
that problem, or to circumvent it without having to install
superlu_dist separately.</p><p class="">Thanks!</p><p class="">Regards,</p><p class="">Roland<br class="">
</p>
<div class="">Am 21.12.2021 um 19:04 schrieb Barry
Smith:<br class="">
</div>
<blockquote type="cite" class="">
<br class="">
<div class=""> I think the problem comes from the following: </div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div class=""><p class="">-DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include
-arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0"</p>
</div>
</blockquote>
which is ok, the location of the CUDA include files is passed in
correctly but when the compile is done inside SuperLU_DIST's
make</div>
<div class=""><br class="">
</div>
<div class="">cd
~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC
&& /usr/local/cuda-11.5/bin/nvcc
-forward-unknown-to-host-compiler -DSUPERLU_DIST_EXPORTS
-Dsuperlu_dist_EXPORTS
-I~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/petsc-build/SRC
-isystem=/usr/local/cuda-11.5/include -DUSE_VENDOR_BLAS
-allow-unsupported-compiler -Xcompiler -fPIC -O3 -ccbin
/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17 -gencode
arch=compute_61,code=sm_61 -Wno-deprecated-gpu-targets
-I/opt/intel/oneapi/mpi/2021.5.0/include -O3 -DNDEBUG
--generate-code=arch=compute_61,code=[compute_61,sm_61]
-Xcompiler=-fPIC -std=c++11 -MD -MT
SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o -MF
CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o.d -x cu -dc
~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/<a href="http://superlu_gpu_utils.cu/" target="_blank" class="">superlu_gpu_utils.cu</a>
-o CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o</div>
<div class=""><br class="">
</div>
<div class="">The </div>
<div class=""><br class="">
</div>
<div class=""> -I/usr/local/cuda-11.5/include</div>
<div class=""><br class="">
</div>
<div class="">has been replaced with </div>
<div class=""><br class="">
</div>
<div class="">-isystem=/usr/local/cuda-11.5/include </div>
<div class=""><br class="">
</div>
<div class="">and I suspect the nvcc doesn't understand it so it ignores
it. I assume somewhere inside the CMAKE processing it is
replacing the -I with the -isystem=</div>
<div class=""><br class="">
</div>
<div class="">I don't know how one could fix this. I suggest you try
installing SuperLU_DIST yourself directly. But I suspect the
same problem will appear.</div>
<div class=""><br class="">
</div>
<div class="">It is possible that if you do not use the Intel compilers but
use for example the GNU compilers the problem may not occur
because cmake may not make the transformation.</div>
<div class=""><br class="">
</div>
<div class="">Barry</div>
<div class=""><br class="">
</div>
<div class="">Googling cmake isystem produces many messages about issues
related to isystem and cmake but it would take a life-time to
understand them.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 21, 2021, at 2:24 AM, Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank" class="">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="">
<div class="">
<div class=""><p class="">I added/replaced all six lines which are in
that commit, getting <br class="">
</p><p class="">Executing: /usr/bin/cmake ..
-DCMAKE_INSTALL_PREFIX=/opt/petsc
-DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib"
-DCMAKE_INSTALL_LIBDIR:STRING="lib"
-DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_BUILD_TYPE=Release
-DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
-DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
-DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib
-DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3 -fPIC
-fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2
-march=native -O3 -fPIC -fopenmp"
-DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native -O3
-fPIC -fopenmp"
-DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
-DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
-DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3
-fopenmp -fPIC -std=gnu++17 -fopenmp"
-DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native -O3
-fopenmp -fPIC -std=gnu++17 -fopenmp"
-DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2 -march=native
-O3 -fopenmp -fPIC -std=gnu++17 -fopenmp"
-DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
-DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
-DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native -O3
-fPIC -fopenmp -fallow-argument-mismatch"
-DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2 -march=native
-O3 -fPIC -fopenmp -fallow-argument-mismatch"
-DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2
-march=native -O3 -fPIC -fopenmp
-fallow-argument-mismatch"
-DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp"
-DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
-DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas
-lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda"
-DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include
-arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0"
-DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler
-Xcompiler -fPIC -O3 -ccbin
/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx -std=c++17
-gencode arch=compute_61,code=sm_61
-Wno-deprecated-gpu-targets
-I"/opt/intel/oneapi/mpi/2021.5.0/include" "
-DUSE_XSDK_DEFAULTS=YES -DTPL_BLAS_LIBRARIES="-lopenblas
-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas
-lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++
-ldl
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-L/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi
-lrt -lpthread -lgfortran -lm
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
-L/opt/intel/oneapi/vpl/2022.0.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran
-lm -lgcc_s -lquadmath"
-DTPL_LAPACK_LIBRARIES="-lopenblas
-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft -lcublas
-lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm -lstdc++
-ldl
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-L/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi
-lrt -lpthread -lgfortran -lm
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
-L/opt/intel/oneapi/vpl/2022.0.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib -lgfortran
-lm -lgcc_s -lquadmath" -Denable_parmetislib=FALSE
-DTPL_ENABLE_PARMETISLIB=FALSE -DXSDK_ENABLE_Fortran=ON
-Denable_tests=0 -Denable_examples=0
-DMPI_C_COMPILE_FLAGS:STRING=""
-DMPI_C_INCLUDE_PATH:STRING=""
-DMPI_C_HEADER_DIR:STRING="" -DMPI_C_LIBRARIES:STRING=""</p><p class="">but the compilation still fails with the same
error.<br class="">
</p>
<div class="">Am 20.12.21 um 22:46 schrieb
Roland Richter:<br class="">
</div>
<blockquote type="cite" class=""><p class="">Yes, just checked, I only included the
changes above the comment...</p><p class="">Will test tomorrow, thanks for the help!</p><p class="">Regards,</p><p class="">Roland<br class="">
</p>
<div class="">Am 20.12.2021 um 21:46
schrieb Barry Smith:<br class="">
</div>
<blockquote type="cite" class="">
<div class=""> Hmm, the fix should now be supplying a
-DCMAKE_CUDA_FLAGS to this cmake command (line 48
at <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4635/diffs" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/4635/diffs</a>)
I do not see that flag being set inside the
configure.log so I am guessing you didn't get the
complete fix.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Executing: /usr/bin/cmake ..
-DCMAKE_INSTALL_PREFIX=/opt/petsc
-DCMAKE_INSTALL_NAME_DIR:STRING="/opt/petsc/lib"
-DCMAKE_INSTALL_LIBDIR:STRING="lib"
-DCMAKE_VERBOSE_MAKEFILE=1
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
-DMPI_C_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc"
-DCMAKE_AR=/usr/bin/ar
-DCMAKE_RANLIB=/usr/bin/ranlib
-DCMAKE_C_FLAGS:STRING="-mavx2 -march=native -O3
-fPIC -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-mavx2
-march=native -O3 -fPIC -fopenmp"
-DCMAKE_C_FLAGS_RELEASE:STRING="-mavx2 -march=native
-O3 -fPIC -fopenmp"
-DCMAKE_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
-DMPI_CXX_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx"
-DCMAKE_CXX_FLAGS:STRING="-mavx2 -march=native -O3
-fopenmp -fPIC -std=gnu++17 -fopenmp"
-DCMAKE_CXX_FLAGS_DEBUG:STRING="-mavx2 -march=native
-O3 -fopenmp -fPIC -std=gnu++17 -fopenmp"
-DCMAKE_CXX_FLAGS_RELEASE:STRING="-mavx2
-march=native -O3 -fopenmp -fPIC -std=gnu++17
-fopenmp"
-DCMAKE_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
-DMPI_Fortran_COMPILER="/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc"
-DCMAKE_Fortran_FLAGS:STRING="-mavx2 -march=native
-O3 -fPIC -fopenmp -fallow-argument-mismatch"
-DCMAKE_Fortran_FLAGS_DEBUG:STRING="-mavx2
-march=native -O3 -fPIC -fopenmp
-fallow-argument-mismatch"
-DCMAKE_Fortran_FLAGS_RELEASE:STRING="-mavx2
-march=native -O3 -fPIC -fopenmp
-fallow-argument-mismatch"
-DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp -fopenmp"
-DBUILD_SHARED_LIBS:BOOL=ON
-DTPL_ENABLE_CUDALIB=TRUE
-DTPL_CUDA_LIBRARIES="-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft
-lcublas -lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda"
-DCUDA_ARCH_FLAGS="-I/usr/local/cuda-11.5/include
-arch=sm_61 -DDEBUGlevel=0 -DPRNTlevel=0"
-DUSE_XSDK_DEFAULTS=YES
-DTPL_BLAS_LIBRARIES="-lopenblas
-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft
-lcublas -lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm
-lstdc++ -ldl
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-L/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi
-lrt -lpthread -lgfortran -lm
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
-L/opt/intel/oneapi/vpl/2022.0.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-lgfortran -lm -lgcc_s -lquadmath"
-DTPL_LAPACK_LIBRARIES="-lopenblas
-Wl,-rpath,/usr/local/cuda-11.5/lib64
-L/usr/local/cuda-11.5/lib64 -lcudart -lcufft
-lcublas -lcusparse -lcusolver -lcurand
-L/usr/local/cuda-11.5/lib64/stubs -lcuda -lm
-lstdc++ -ldl
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-L/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-L/opt/intel/oneapi/mpi/2021.5.0/lib -lmpifort -lmpi
-lrt -lpthread -lgfortran -lm
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2022.0.0/lib
-L/opt/intel/oneapi/vpl/2022.0.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.5.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.5.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-L/opt/intel/oneapi/ipp/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2022.0.1/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-L/opt/intel/oneapi/dal/2021.5.1/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-L/opt/intel/oneapi/compiler/2022.0.1/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.5.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.5.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.5.0/lib
-lgfortran -lm -lgcc_s -lquadmath"
-Denable_parmetislib=FALSE
-DTPL_ENABLE_PARMETISLIB=FALSE
-DXSDK_ENABLE_Fortran=ON -Denable_tests=0
-Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING=""
-DMPI_C_INCLUDE_PATH:STRING=""
-DMPI_C_HEADER_DIR:STRING=""
-DMPI_C_LIBRARIES:STRING=""</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 20, 2021, at 2:59 PM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank" class="">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="">
<div class="">
<div class=""><p class="">I introduced the changes from that
patch directly, without checking out. Is
that insufficient?</p><p class="">Regards,</p><p class="">Roland<br class="">
</p>
<div class="">Am 20.12.2021 um
20:38 schrieb Barry Smith:<br class="">
</div>
<blockquote type="cite" class="">
<div class=""><br class="">
</div>
Are you sure you have the correct PETSc
branch? From configure.log it has
<div class=""><br class="">
</div>
<div class="">
<div class=""> Defined
"VERSION_GIT" to
""v3.16.2-466-g959e1fce86""</div>
<div class=""> Defined
"VERSION_DATE_GIT" to ""2021-12-18
11:17:24 -0600""</div>
<div class=""> Defined
"VERSION_BRANCH_GIT" to ""master""</div>
<div class=""><br class="">
</div>
<div class="">It should have
balay/slu-without-omp-3 for the branch.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 20, 2021, at
10:50 AM, Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank" class="">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="">
<div class="">
<div class=""><p class="">In that case it fails
with</p><p class=""><i class="">~/Downloads/git-files/petsc/mpich-complex-linux-gcc-demo/externalpackages/git.superlu_dist/SRC/cublas_utils.h:22:10:
fatal error: cublas_v2.h: No
such file or directory</i></p><p class="">even though this
header is available. I assume
some header paths are not set
correctly?</p><p class="">Thanks,</p><p class="">regards,</p><p class="">Roland<br class="">
</p>
<div class="">Am
20.12.21 um 16:29 schrieb Barry
Smith:<br class="">
</div>
<blockquote type="cite" class="">
<div class=""><br class="">
</div>
Please try the
branch balay/slu-without-omp-3
It is in MR <a href="https://gitlab.com/petsc/petsc/-/merge_requests/4635" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/4635</a>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 20,
2021, at 8:14 AM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank" class="">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="">
<div class="">
<div class=""><p class="">Hei,</p><p class="">I tried to
combine CUDA with
superlu_dist in
petsc using the
following
configure-line:</p><p class=""><i class="">./configure
PETSC_ARCH=mpich-complex-linux-gcc-demo
--CC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicc
--CXX=/opt/intel/oneapi/mpi/2021.5.0/bin/mpicxx
--FC=/opt/intel/oneapi/mpi/2021.5.0/bin/mpifc --CFLAGS="-mavx2
-march=native -O3"
--CXXFLAGS="-mavx2
-march=native -O3"
--FFLAGS="-mavx2
-march=native -O3"
--CUDAFLAGS=-allow-unsupported-compiler --CUDA-CXX=g++
--prefix=/opt/petsc
--with-blaslapack=1 --with-mpi=1 --with-scalar-type=complex
--download-suitesparse=1
--with-cuda
--with-debugging=0
--with-openmp
--download-superlu_dist
--force</i></p><p class="">but the
configure-step fails
with several errors
correlated with CUDA
and superlu_dist,
the first one being</p><p class=""><i class="">cublas_utils.c:21:37:
error:
‘CUDART_VERSION’
undeclared (first
use in this
function); did you
mean
‘CUDA_VERSION’?</i><i class=""><br class="">
</i><i class="">
21 |
printf("CUDA
version: v
%d\n",CUDART_VERSION);</i><i class=""><br class="">
</i><i class="">
| ^~~~~~~~~~~~~~</i><i class=""><br class="">
</i><i class="">
| CUDA_VERSION</i></p><p class="">Compiling
superlu_dist
separately works,
though (including
CUDA).</p><p class="">Is there a
bug somewhere in the
configure-routine? I
attached the full
configure-log.<br class="">
</p><p class="">Thanks!</p><p class="">Regards,</p><p class="">Roland<br class="">
</p>
</div>
<span id="gmail-m_-6005507413301060485cid:F1E9B5DB-A92E-4C95-83FD-3F462C695FE9" class=""><configure.log></span></div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
<span id="gmail-m_-6005507413301060485cid:4A36F88D-58F8-40D3-A642-41C361B045C8" class=""><configure.log></span></div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
</blockquote>
</div>
<span id="gmail-m_-6005507413301060485cid:16632CC5-01A0-4F67-AA92-6F8BEBEE763B" class=""><configure.log></span></div>
</blockquote>
</div>
<br class="">
</blockquote>
</div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>