<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Junchao’s suggestion was on the money it seems.<div class=""><br class=""></div><div class="">I played around trying to build petsc with various compilers (xl, pgi, gcc, clang), and a few version and they all had issues except when using gcc and clang. As it turns out the gcc that nvcc falls back on on Lassen is positively geriatric, I believe gcc v4.9.3 as it is the default gcc loaded. -std=c++14 wasn’t introduced to gcc until v5.2. This gcc is only upgraded automatically if one uses the gcc or clang compiler modules (as these will override “gcc” either via replacement or alias). Using -ccbin to override the default compiler and set it to mpicc worked.</div><div class=""><br class=""></div><div class="">So to sum up, configure checked mpicc (= xl v2020.11.12) and found correctly that it supported c++14, and so assumed that nvcc’s host compiler also supported this, adding it to configure arguments. But this wasn’t the case since nvcc silently defaults to gcc, which in this case was too old. I also tried looking for a way to figure out exactly what host compiler nvcc is using, but so far no luck.</div><div class=""><br class=""></div><div class="">Note that Lassen also does some funky silent argument additions, which you can see if you run “nvcc -vvvv”, but I don’t think these affected configure (since the c++14 flag comes from Junchao’s additions in cuda.py):</div><div class=""><br class=""></div><div class="">Setting default C++ std via -std=c++14 to match g++ default <—————— Silently does this if your “g++” supports -std=c++14<br class="">Set TMPDIR to /var/tmp/faibuss to workaround name conflicts<br class="">nvidia-wrapper executing after fixup:<br class="">export LLNL_CALLED_FROM_NVCC=11.1.1<br class="">+ exec /usr/tce/packages/cuda/cuda-11.1.1/nvidia/bin/nvcc -std=c++14 -Xlinker '"-rpath=/usr/tce/packages/cuda/cuda-11.1.1/nvidia/lib64:/usr/tce/packages/cuda/cuda-11.1.1"'<br class="">nvcc fatal : No input files specified; use option --help for more information</div><div class=""><div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Dec 13, 2020, at 18:32, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Jacob,</div><div class=""> Do you need to add 'CUDAFLAGS=-ccbin xlc++' to specify the host compiler for CUDA? Note in cuda.py I added</div><div class=""><br class=""></div><div class=""><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; line-height: 21px; white-space: pre;" class=""><div class=""> <span style="color:rgb(175,0,219)" class="">if</span> <span style="color:rgb(0,0,255)" class="">self</span>.compilers.cxxdialect <span style="color:rgb(0,0,255)" class="">in</span> [<span style="color:rgb(163,21,21)" class="">'C++11'</span>,<span style="color:rgb(163,21,21)" class="">'C++14'</span>]: <span style="color:rgb(0,128,0)" class="">#nvcc is a C++ compiler so it is always good to add -std=xxx. It is even crucial when using thrust complex (see MR 2822)</span></div><div class=""> <span style="color:rgb(0,0,255)" class="">self</span>.setCompilers.CUDAFLAGS += <span style="color:rgb(163,21,21)" class="">' -std='</span> + <span style="color:rgb(0,0,255)" class="">self</span>.compilers.cxxdialect.lower()</div></div></div><div class=""><br class=""></div><div class=""> In your configure.log, there are </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px" class=""><div class="">#define PETSC_HAVE_CXX_DIALECT_CXX11 1</div><div class="">#define PETSC_HAVE_CXX_DIALECT_CXX14 1</div></blockquote><div class=""><br class=""></div><div class="">I guess without -ccbin, nvcc uses gcc by default and your gcc does not support C++14.</div><div class=""><br class=""></div><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Dec 13, 2020 at 1:25 PM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Hello All,<div class=""><br class=""></div><div class="">Does anyone have any experience building petsc with cuda support on Lassen? I’ve been having trouble building with ibm xl compilers + spectrum-mpi + nvcc. NVCC seems to not like -std=c++14 argument, complaining that its configured host compiler doesn’t support it, but compiling the following “<a href="http://test.cc/" target="_blank" class="">test.cc</a>":</div><div class=""><br class=""></div><div class="">#include <stdlib.h><br class=""><br class="">int main(int argc, char **argv)<br class="">{ <br class=""> int i = 1;<br class=""> i += argc;<br class=""> return(i);<br class="">}<br class=""><br class=""></div><div class="">With mpicc -std=c++14 <a href="http://test.cc/" target="_blank" class="">test.cc</a> produces zero errors. </div><div class="">------------------------------------------------------------------------</div><div class=""><br class=""></div><div class="">Modules loaded:</div><div class=""><br class=""></div><div class="">module load xl/2020.11.12-cuda-11.1.1 <br class="">module load spectrum-mpi<br class="">module load cuda/11.1.1<br class="">module load python/3.8.2<br class="">module load cmake<br class="">module load valgrind<br class="">module load lapack<br class=""><br class=""></div><div class="">My configure commands:</div><div class=""><br class=""></div><div class="">./configure --with-cc=mpicc --with-cxx=mpiCC --with-fc=mpifort --with-cuda --with-debugging=1 PETSC_ARCH=arch-linux-c-debug</div><div class=""><br class=""></div><div class="">The error:</div><div class=""><br class=""></div><div class="">TESTING: findMPIInc from config.packages.MPI(config/BuildSystem/config/packages/MPI.py:636) *******************************************************************************<br class=""> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):<br class="">-------------------------------------------------------------------------------<br class="">Bad compiler flag: -I/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/include<br class="">*******************************************************************************</div><div class=""><br class=""></div><div class="">The actual configure.log error:</div><div class=""><br class=""></div><div class="">Executing: nvcc -c -o /var/tmp/petsc-2v0k4k61/config.setCompilers/conftest.o -I/var/tmp/petsc-2v0k4\<br class="">k61/config.setCompilers -I/var/tmp/petsc-2v0k4k61/config.types -g -std=c++14 -I/usr/tce/packages/s\<br class="">pectrum-mpi/ibm/spectrum-mpi-rolling-release/include -Wno-deprecated-gpu-targets /var/tmp/petsc-2v\<br class="">0k4k61/config.setCompilers/<a href="http://conftest.cu/" target="_blank" class="">conftest.cu</a> <br class="">Possible ERROR while running compiler:<br class="">stderr:<br class="">nvcc warning : The -std=c++14 flag is not supported with the configured host compiler. Flag will be\<br class=""> ignored.<br class="">Source:<br class="">#include "confdefs.h"<br class="">#include "conffix.h"<br class=""><br class="">int main() {<br class="">;<br class=""> return 0;<br class="">}<br class=""> Rejecting compiler flag -I/usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/include due to <br class="">nvcc warning : The -std=c++14 flag is not supported with the configured host compiler. Flag will be ignored.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class="">Cell: (312) 694-3391</div></div>
</div>
</div></div><div style="overflow-wrap: break-word;" class=""><div class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></body></html>