[petsc-dev] I am getting this error ...

Mon Nov 8 11:36:58 CST 2021

Better yet.. [from Junchao]

export CRAY_ACCEL_TARGET=host

Satish

On Fri, 5 Nov 2021, Mark Adams wrote:

> Bingo!
> 
> On Fri, Nov 5, 2021 at 12:54 PM Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > Yeah remove [C,CPP,CXX,CXXPP,F] FLAGS
> >
> > CPP defaults to '$CC -E' - so with "--with-cc='cc -mp=gpu'" - it should
> > use "cc -mp=gpu -E"
> >
> > Satish
> >
> > On Fri, 5 Nov 2021, Mark Adams wrote:
> >
> > > How about CPP flags?
> > >
> > > On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <balay at mcs.anl.gov> wrote:
> > >
> > > > I guess another way to deal with this is: not use CFLAGS etc..
> > > >
> > > >
> > > > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu'
> > > >
> > > > Satish
> > > >
> > > >
> > > > On Fri, 5 Nov 2021, Mark Adams wrote:
> > > >
> > > > > Yes, thanks.
> > > > > I emailed the NERSc person and told him where we are and that we
> > could
> > > > fix
> > > > > this manually, but I don't have this issue on Summit ...
> > > > > He did not understand for the longest time, not sure he does now,
> > that we
> > > > > do not add -gpu.
> > > > > And I gave him your reproducer.
> > > > > With any luck, with the reproducer, they can figure this out at some
> > > > point.
> > > > > MPI is not working well for my scaling studies and my app has also
> > given
> > > > up
> > > > > on nvhpc, so we are good.
> > > > >
> > > > >
> > > > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <balay at mcs.anl.gov>
> > wrote:
> > > > >
> > > > > > For now - you could manually edit petscvariables and remove
> > -mp=gpu
> > > > from
> > > > > > it.
> > > > > >
> > > > > > Its primarily required to make configure happy.
> > > > > >
> > > > > > Satish
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, 4 Nov 2021, Barry Smith wrote:
> > > > > >
> > > > > > >
> > > > > > >    This comes from the persistent problem with PETSc's make
> > system
> > > > using
> > > > > > too many flags for compiling CUDA that have not been tested by
> > > > configure.
> > > > > > See below the -mp=gpu is provided probably from the CPPFLAGS or
> > > > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to
> > > > compile
> > > > > > CUDA code!
> > > > > > >
> > > > > > >
> > > > > > > Using CUDA compile:
> > > > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o
> > > > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include
> > > > -I/opt/cray/pe/libsci/
> > > > > > 21.08.1.2/NVIDIA/20.7/x86_64/include
> > -I/opt/cray/pe/pmi/6.0.14/include
> > > > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include
> > > > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include  -g
> > > > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10
> > > > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode
> > > > > > arch=compute_80,code=sm_80  -Wno-deprecated-gpu-targets  -c
> > > > > > --compiler-options=-I/global/homes/m/madams/petsc/include
> > > > > >
> > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include
> > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu
> > > > > > -I/global/homes/m/madams/petsc/include
> > > > > >
> > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include
> > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfadams at lbl.gov>
> > wrote:
> > > > > > > >
> > > > > > > > OK, configure is done.
> > > > > > > > Maybe I have too many -mp=gpu
> > > > > > > >
> > > > > > > >        CUDAC
> > > > > >
> > > >
> > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o
> > > > > > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you
> > > > mean
> > > > > > ‘-mpku’?
> > > > > > > >
> > > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsmith at petsc.dev
> > > > <mailto:
> > > > > > bsmith at petsc.dev>> wrote:
> > > > > > > >
> > > > > > > >   Need the same thing for the C++ preprocessor flag
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfadams at lbl.gov
> > <mailto:
> > > > > > mfadams at lbl.gov>> wrote:
> > > > > > > >>
> > > > > > > >> It gets a lot further.
> > > > > > > >>
> > > > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfadams at lbl.gov
> > > > <mailto:
> > > > > > mfadams at lbl.gov>> wrote:
> > > > > > > >> OK, sorry I missed the CPPFLAGS. It is running now.
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay <
> > balay at mcs.anl.gov
> > > > > > <mailto:balay at mcs.anl.gov>> wrote:
> > > > > > > >> Multiple e-mail threads on the same issue (:
> > > > > > > >>
> > > > > > > >> As suggested in my earlier thread - add -mp=gpu to both
> > CPPFLAGS
> > > > and
> > > > > > CFLAGS [or LDFLAGS]
> > > > > > > >>
> > > > > > > >> Satish
> > > > > > > >>
> > > > > > > >> -------
> > > > > > > >> Executing: cc  -o
> > /tmp/petsc-Vvs8_T/config.setCompilers/conftest
> > > >  -g
> > > > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4
> > > > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o
> > > > > > > >> Possible ERROR while running linker:
> > > > > > > >> stderr:
> > > > > > > >> nvc-Warning-The -gpu option has no effect unless a
> > > > language-specific
> > > > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu,
> > > > -stdpar,
> > > > > > -cuda)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote:
> > > > > > > >>
> > > > > > > >> > It is CPPFLAGS. I seem to get the same behavior.
> > > > > > > >> >
> > > > > > > >> > FWIW, I did get this response from NERSc but I don't know
> > how to
> > > > > > interpret
> > > > > > > >> > it.
> > > > > > > >> >
> > > > > > > >> > He seems to be saying that I don't need -mp=gpu for the
> > device
> > > > > > compiler
> > > > > > > >> > (nvcc). He seems to think that I am adding -gpu.
> > > > > > > >> >
> > > > > > > >> > nvcc -- The device compiler does not need any of those flags
> > > > > > because it
> > > > > > > >> > already knows that it's being fed cuda code. The warning
> > you're
> > > > > > seeing is
> > > > > > > >> > coming from nvc (which is the host / CPU side compiler) if
> > > > you're
> > > > > > in the
> > > > > > > >> > PrgEnv-nvidia environment. You should not need to add
> > -mp=gpu
> > > > and
> > > > > > -cuda,
> > > > > > > >> > please just add the -cuda flag (to your host code) not to
> > the
> > > > > > device code.
> > > > > > > >> >
> > > > > > > >> > I will try to talk with this guy again.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith <
> > bsmith at petsc.dev
> > > > > > <mailto:bsmith at petsc.dev>> wrote:
> > > > > > > >> >
> > > > > > > >> > >
> > > > > > > >> > >   Yes, you need to use the CPPFLAGS which maybe called
> > > > CPPCFLAGS
> > > > > > I am not
> > > > > > > >> > > sure
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfadams at lbl.gov
> > > > <mailto:
> > > > > > mfadams at lbl.gov>> wrote:
> > > > > > > >> > >
> > > > > > > >> > > Ah, CCFLAGS does not seem to work.
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith <
> > bsmith at petsc.dev
> > > > > > <mailto:bsmith at petsc.dev>> wrote:
> > > > > > > >> > >
> > > > > > > >> > >>
> > > > > > > >> > >>   You have to pass in the flag to turn off the bitching
> > about
> > > > > > -gpu to the
> > > > > > > >> > >> C preprocessor, not the C compiler.
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> stderr:
> > > > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a
> > > > > > language-specific
> > > > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc,
> > > > > > -mp=gpu, -stdpar,
> > > > > > > >> > >> -cuda)
> > > > > > > >> > >> Source:
> > > > > > > >> > >> #include "confdefs.h"
> > > > > > > >> > >> #include "conffix.h"
> > > > > > > >> > >> #include <stdlib.h>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams <
> > mfadams at lbl.gov
> > > > > > <mailto:mfadams at lbl.gov>> wrote:
> > > > > > > >> > >> >
> > > > > > > >> > >> > on Perlmutter with nvhpc:
> > > > > > > >> > >> >
> > > > > > > >> > >> >               Defined make macro "CPP" to "cc --use
> > cpp32"
> > > > > > > >> > >> > Preprocessing source:
> > > > > > > >> > >> > #include "confdefs.h"
> > > > > > > >> > >> > #include "conffix.h"
> > > > > > > >> > >> > #include <stdlib.h>
> > > > > > > >> > >> >
> > > > > > > >> > >> > Executing: cc --use cpp32
> > > > > > -I/tmp/petsc-jV9U1b/config.setCompilers
> > > > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c
> > > > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1
> > > > > > > >> > >> > stderr:
> > > > > > > >> > >> > nvc-Error-Unknown switch: --use
> > > > > > > >> > >> > Source:
> > > > > > > >> > >> > <configure.log>
> > > > > > > >> > >>
> > > > > > > >> > >> <configure.log>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> <configure.log>
> > > > > > > >
> > > > > > > > <make.log><configure.log>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 