<div dir="ltr">Bingo!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 5, 2021 at 12:54 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Yeah remove [C,CPP,CXX,CXXPP,F] FLAGS<br>
<br>
CPP defaults to '$CC -E' - so with "--with-cc='cc -mp=gpu'" - it should use "cc -mp=gpu -E"<br>
<br>
Satish<br>
<br>
On Fri, 5 Nov 2021, Mark Adams wrote:<br>
<br>
> How about CPP flags?<br>
> <br>
> On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> <br>
> > I guess another way to deal with this is: not use CFLAGS etc..<br>
> ><br>
> ><br>
> > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu'<br>
> ><br>
> > Satish<br>
> ><br>
> ><br>
> > On Fri, 5 Nov 2021, Mark Adams wrote:<br>
> ><br>
> > > Yes, thanks.<br>
> > > I emailed the NERSc person and told him where we are and that we could<br>
> > fix<br>
> > > this manually, but I don't have this issue on Summit ...<br>
> > > He did not understand for the longest time, not sure he does now, that we<br>
> > > do not add -gpu.<br>
> > > And I gave him your reproducer.<br>
> > > With any luck, with the reproducer, they can figure this out at some<br>
> > point.<br>
> > > MPI is not working well for my scaling studies and my app has also given<br>
> > up<br>
> > > on nvhpc, so we are good.<br>
> > ><br>
> > ><br>
> > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > > > For now - you could manually edit petscvariables and remove -mp=gpu<br>
> > from<br>
> > > > it.<br>
> > > ><br>
> > > > Its primarily required to make configure happy.<br>
> > > ><br>
> > > > Satish<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On Thu, 4 Nov 2021, Barry Smith wrote:<br>
> > > ><br>
> > > > ><br>
> > > > > This comes from the persistent problem with PETSc's make system<br>
> > using<br>
> > > > too many flags for compiling CUDA that have not been tested by<br>
> > configure.<br>
> > > > See below the -mp=gpu is provided probably from the CPPFLAGS or<br>
> > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to<br>
> > compile<br>
> > > > CUDA code!<br>
> > > > ><br>
> > > > ><br>
> > > > > Using CUDA compile:<br>
> > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o<br>
> > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include<br>
> > -I/opt/cray/pe/libsci/<br>
> > > > <a href="http://21.08.1.2/NVIDIA/20.7/x86_64/include" rel="noreferrer" target="_blank">21.08.1.2/NVIDIA/20.7/x86_64/include</a> -I/opt/cray/pe/pmi/6.0.14/include<br>
> > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include<br>
> > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g<br>
> > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10<br>
> > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode<br>
> > > > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c<br>
> > > > --compiler-options=-I/global/homes/m/madams/petsc/include<br>
> > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include<br>
> > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu<br>
> > > > -I/global/homes/m/madams/petsc/include<br>
> > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include<br>
> > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> > > > > ><br>
> > > > > > OK, configure is done.<br>
> > > > > > Maybe I have too many -mp=gpu<br>
> > > > > ><br>
> > > > > > CUDAC<br>
> > > ><br>
> > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o<br>
> > > > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you<br>
> > mean<br>
> > > > ‘-mpku’?<br>
> > > > > ><br>
> > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a><br>
> > <mailto:<br>
> > > > <a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>> wrote:<br>
> > > > > ><br>
> > > > > > Need the same thing for the C++ preprocessor flag<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a> <mailto:<br>
> > > > <a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
> > > > > >><br>
> > > > > >> It gets a lot further.<br>
> > > > > >><br>
> > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><br>
> > <mailto:<br>
> > > > <a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
> > > > > >> OK, sorry I missed the CPPFLAGS. It is running now.<br>
> > > > > >> Thanks,<br>
> > > > > >><br>
> > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a><br>
> > > > <mailto:<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>>> wrote:<br>
> > > > > >> Multiple e-mail threads on the same issue (:<br>
> > > > > >><br>
> > > > > >> As suggested in my earlier thread - add -mp=gpu to both CPPFLAGS<br>
> > and<br>
> > > > CFLAGS [or LDFLAGS]<br>
> > > > > >><br>
> > > > > >> Satish<br>
> > > > > >><br>
> > > > > >> -------<br>
> > > > > >> Executing: cc -o /tmp/petsc-Vvs8_T/config.setCompilers/conftest<br>
> > -g<br>
> > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4<br>
> > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o<br>
> > > > > >> Possible ERROR while running linker:<br>
> > > > > >> stderr:<br>
> > > > > >> nvc-Warning-The -gpu option has no effect unless a<br>
> > language-specific<br>
> > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu,<br>
> > -stdpar,<br>
> > > > -cuda)<br>
> > > > > >><br>
> > > > > >><br>
> > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote:<br>
> > > > > >><br>
> > > > > >> > It is CPPFLAGS. I seem to get the same behavior.<br>
> > > > > >> ><br>
> > > > > >> > FWIW, I did get this response from NERSc but I don't know how to<br>
> > > > interpret<br>
> > > > > >> > it.<br>
> > > > > >> ><br>
> > > > > >> > He seems to be saying that I don't need -mp=gpu for the device<br>
> > > > compiler<br>
> > > > > >> > (nvcc). He seems to think that I am adding -gpu.<br>
> > > > > >> ><br>
> > > > > >> > nvcc -- The device compiler does not need any of those flags<br>
> > > > because it<br>
> > > > > >> > already knows that it's being fed cuda code. The warning you're<br>
> > > > seeing is<br>
> > > > > >> > coming from nvc (which is the host / CPU side compiler) if<br>
> > you're<br>
> > > > in the<br>
> > > > > >> > PrgEnv-nvidia environment. You should not need to add -mp=gpu<br>
> > and<br>
> > > > -cuda,<br>
> > > > > >> > please just add the -cuda flag (to your host code) not to the<br>
> > > > device code.<br>
> > > > > >> ><br>
> > > > > >> > I will try to talk with this guy again.<br>
> > > > > >> ><br>
> > > > > >> > Thanks,<br>
> > > > > >> ><br>
> > > > > >> ><br>
> > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a><br>
> > > > <mailto:<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>> wrote:<br>
> > > > > >> ><br>
> > > > > >> > ><br>
> > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called<br>
> > CPPCFLAGS<br>
> > > > I am not<br>
> > > > > >> > > sure<br>
> > > > > >> > ><br>
> > > > > >> > ><br>
> > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><br>
> > <mailto:<br>
> > > > <a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
> > > > > >> > ><br>
> > > > > >> > > Ah, CCFLAGS does not seem to work.<br>
> > > > > >> > ><br>
> > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a><br>
> > > > <mailto:<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>> wrote:<br>
> > > > > >> > ><br>
> > > > > >> > >><br>
> > > > > >> > >> You have to pass in the flag to turn off the bitching about<br>
> > > > -gpu to the<br>
> > > > > >> > >> C preprocessor, not the C compiler.<br>
> > > > > >> > >><br>
> > > > > >> > >><br>
> > > > > >> > >> stderr:<br>
> > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a<br>
> > > > language-specific<br>
> > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc,<br>
> > > > -mp=gpu, -stdpar,<br>
> > > > > >> > >> -cuda)<br>
> > > > > >> > >> Source:<br>
> > > > > >> > >> #include "confdefs.h"<br>
> > > > > >> > >> #include "conffix.h"<br>
> > > > > >> > >> #include <stdlib.h><br>
> > > > > >> > >><br>
> > > > > >> > >><br>
> > > > > >> > >><br>
> > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><br>
> > > > <mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
> > > > > >> > >> ><br>
> > > > > >> > >> > on Perlmutter with nvhpc:<br>
> > > > > >> > >> ><br>
> > > > > >> > >> > Defined make macro "CPP" to "cc --use cpp32"<br>
> > > > > >> > >> > Preprocessing source:<br>
> > > > > >> > >> > #include "confdefs.h"<br>
> > > > > >> > >> > #include "conffix.h"<br>
> > > > > >> > >> > #include <stdlib.h><br>
> > > > > >> > >> ><br>
> > > > > >> > >> > Executing: cc --use cpp32<br>
> > > > -I/tmp/petsc-jV9U1b/config.setCompilers<br>
> > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c<br>
> > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1<br>
> > > > > >> > >> > stderr:<br>
> > > > > >> > >> > nvc-Error-Unknown switch: --use<br>
> > > > > >> > >> > Source:<br>
> > > > > >> > >> > <configure.log><br>
> > > > > >> > >><br>
> > > > > >> > >> <configure.log><br>
> > > > > >> > ><br>
> > > > > >> > ><br>
> > > > > >> > ><br>
> > > > > >> ><br>
> > > > > >><br>
> > > > > >> <configure.log><br>
> > > > > ><br>
> > > > > > <make.log><configure.log><br>
> > > > ><br>
> > > > ><br>
> > > ><br>
> > ><br>
> ><br>
> <br>
</blockquote></div>