[petsc-users] unable to build petsc with cuda / superlu-dist enabled
byron
lbgpublic at gmail.com
Fri Aug 26 11:13:35 CDT 2022
It looks like I've (we've) cracked it, at least the petsc stage of what I'm
trying to do. To get the configure to work I used the following
python3 ./configure --download-hypre=1 --with-shared-libraries=1
--download-hdf5=1 --download-hdf5-fortran-bindings=0 --with-debugging=no
--download-fblaslapack=1 --download-metis=1 --download-ptscotch=1
--download-parmetis=1 --download-superlu_dist=1 --download-mumps=1
--download-strumpack=1 --download-scalapack=1 --download-slepc=1
--with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11
-with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-cuda
-CUDAOPTFLAGS="" --with-cuda-dialect=cxx11
--prefix=/home/lgoodman/Projects/petscExamples/main-build-CUDAOPTFLAGS-zero-cxx-dialect-unspc-cuda-dial-11/installation
Where -CUDAOPTFLAGS="" was the solutions to getting rid of the duplicate
-O3
then I was getting errors about 'redefinition of argument 'std'" so i added
--with-cxx-dialect=C++11
and
--with-cuda-dialect=cxx11
and that got rid of those errors. I image there are cleaner / more elegant
ways of doing this but I'll settle for this for now.
Thanks a lot for the help guys, much appreciated
Best regards
Byron
On Fri, Aug 26, 2022 at 4:06 PM Barry Smith <bsmith at petsc.dev> wrote:
>
> Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally
> SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and
> these that get added to the one's PETSc passes in (they do not replace
> them). A little further down in the file it displays a compile line with
> nvcc that has two sets of -O3 (as well as two sets of other things as well).
>
> I think we changed the main branch PETSc to not pass down the
> optimization flags to the CMake CUDA to prevent the problem from happening.
> In other words, we let the package decide its optimization flags instead of
> trying to tell it what to use.
>
> Barry
>
> The root of the problem is that compiler APIs are a bunch of strings
> passed on the command line to the compiler without any structure.
>
>
> On Aug 26, 2022, at 9:47 AM, byron <lbgpublic at gmail.com> wrote:
>
> Hi Barry
>
> I dont think I see the same thing.
>
> When I do a build using the release branch petsc passes the following
> arguements to superlu_dist (taken from
> /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf)
> which has only one -O3 entry
>
> -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation
> -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib"
> -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1
> -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc"
> -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar
> -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC
> -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -DDEBUGlevel=0
> -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch
> -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx"
> -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O
> -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g
> -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector
> -g -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0"
> -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90"
> -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp"
> -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0
> -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING="
> -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE
> -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver
> -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75"
> -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14
> -Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75
> -Wno-deprecated-gpu-targets
> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
> -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0"
> -DUSE_XSDK_DEFAULTS=YES
> -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lflapack
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm
> -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib
> -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08
> -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm
> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9
> -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath
> -lpthread"
> -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib"
> -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lparmetis
> -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib
> -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand
> -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0
> -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING=""
> -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING=""
> -DMPI_C_LIBRARIES:STRING=""
>
> Or am I looking at the wrong thing?
>
> I also tried doing the build downloading petsc from the main branch but
> that died at the same point and looked to give the same "redefinition of
> argument 'optimize'" errors
>
> Thanks
> Byron
>
>
>
> On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob.fai at gmail.com>
> wrote:
>
>> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as
>> well.
>>
>> Main should hopefully *also* warn about this, but unless you really
>> really want -std=c++11 or -std=c++14, it is better to leave out this
>> configure flag (--with-cxx-dialect).
>>
>> If --with-cxx-dialect is unspecified (or set to “auto”) configure will
>> pick the most appropriate flag for you, taking your package environment
>> into account when choosing. If you specify the flag configure must assume
>> the user knows something it doesn’t and will blindly take what is given.
>>
>> Best regards,
>>
>> Jacob Faibussowitsch
>> (Jacob Fai - booss - oh - vitch)
>>
>> > On Aug 25, 2022, at 11:05, Barry Smith <bsmith at petsc.dev> wrote:
>> >
>> >
>> > Oh yes, my mistake I misinterpreted the error message. It is actually
>> the nvcc compiler complaining about multiple command line arguments related
>> to optimization (which it is really picky about even if the two arguments
>> are identical.) I reformatted some of the configure output to show this
>> >
>> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode
>> arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> > -DDEBUGlevel=0 -DPRNTlevel=0
>> >
>> > the next fragment is
>> >
>> > -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes
>> to superlu_dist
>> >
>> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0"
>> >
>> > and then presumably, the superlu_dist cmake process adds its own
>> arguments which overlap.
>> >
>> > If I remember correctly this is fixed in the main branch of PETSc,
>> could you do an identical build with the main branch? If that works we will
>> just need to figure out how to get MOOSE to work with main PETSc branch.
>> >
>> > Barry
>> >
>> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as
>> well.
>> >
>> >
>> > --------------------
>> >
>> > [ 15%] Building CUDA object
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o
>> > cd
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> && /usr/bin/nvcc
>> >
>> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS
>> >
>> >
>> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC
>> >
>> > -I/home/blahblah/Projects/220824-petsc-build/installation/include
>> -I/usr/lib/x86_64-linux-gnu/openmpi/lib
>> >
>> > -DUSE_VENDOR_BLAS -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3
>> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets
>> >
>> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi
>> -I/usr/lib/x86_64-linux-gnu/openmpi/include
>> >
>> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG
>> >
>> > --generate-code=arch=compute_75,code=[compute_75,sm_75]
>> >
>> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c
>> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/
>> superlu_gpu_utils.cu -o
>> >
>> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu
>> >
>> >
>> >
>> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpublic at gmail.com> wrote:
>> >>
>> >> Hi Barry
>> >>
>> >> There doesn't appear to be any reference to 'optimize' in the only
>> copy of superlu_gpu_utils.cu I could find, I've attached a copy.
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsmith at petsc.dev> wrote:
>> >>
>> >> nvcc fatal : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >> nvcc fatal : redefinition of argument 'optimize'
>> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499:
>> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1
>> >> make[2]: *** Waiting for unfinished jobs....
>> >>
>> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu
>> Could you check that file for any "redefinition of optimize"? Unfortunately
>> the nvcc compiler isn't kind enough to print the line number of the
>> redefinition or that information is lost somehow in the build process.
>> >>
>> >> Barry
>> >>
>> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpublic at gmail.com> wrote:
>> >>>
>> >>> Hi All
>> >>>
>> >>> I am having trouble getting petsc to build with cuda / superlu-dist
>> enabled. I have been trying to build it as part of another software tool,
>> moose, hence the choice of options being passed to configure
>> >>>
>> >>> python3 ./configure --download-hypre=1 \
>> >>> --with-shared-libraries=1 \
>> >>> --download-hdf5=1 \
>> >>> --download-hdf5-fortran-bindings=0 \
>> >>> --with-debugging=no \
>> >>> --download-fblaslapack=1 \
>> >>> --download-metis=1 \
>> >>> --download-ptscotch=1 \
>> >>> --download-parmetis=1 \
>> >>> --download-superlu_dist=1 \
>> >>> --download-mumps=1 \
>> >>> --download-strumpack=1 \
>> >>> --download-scalapack=1 \
>> >>> --download-slepc=1 \
>> >>> --with-mpi=1 \
>> >>> --with-openmp=1 \
>> >>> --with-cxx-dialect=C++11 \
>> >>> --with-fortran-bindings=0 \
>> >>> --with-sowing=0 \
>> >>> --with-64-bit-indices \
>> >>> --with-cuda \
>> >>> --prefix=/home/blah/blah
>> >>>
>> >>> it fails with
>> >>>
>> >>>
>> =============================================================================================
>>
>> Installing PTScotch; this may take several
>> minutes
>>
>> =============================================================================================
>>
>>
>> =============================================================================================
>>
>> Trying to download git://
>> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST
>>
>>
>> =============================================================================================
>>
>>
>> =============================================================================================
>>
>> Configuring SUPERLU_DIST with cmake; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>> =============================================================================================
>>
>> Compiling and installing SUPERLU_DIST; this
>> may take several minutes
>>
>> =============================================================================================
>>
>>
>> *******************************************************************************
>> >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log
>> for details):
>> >>>
>> -------------------------------------------------------------------------------
>> >>> Error running make on SUPERLU_DIST
>> >>>
>> *******************************************************************************
>> >>>
>> >>> I've attached the configure.log.
>> >>>
>> >>> I've tried a few different things but nothing that resulted in a
>> successfull build. The most promising was I tried to build superlu_dist
>> separately with the configure options that petsc passes it and then kept
>> trying the build over and over droping out different configure options
>> until I got it to work. In that case removing a "-std=c++14" entry
>> appeared to fix the problem but then when I made the same change in how
>> petsc builds superlu-dist the make failed on superlu-dist with a different
>> error. I get the feeling I'm dealing with symptoms of the problem rather
>> than the cause.
>> >>>
>> >>> Thanks
>> >>> Byron
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <configure.log>
>> >>
>> >> <superlu_gpu_utils.cu>
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220826/6730cf0f/attachment-0001.html>
More information about the petsc-users
mailing list