<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 16, 2021, at 7:37 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Remove --with-kokkos-cuda-arch=VOLTA70 from your configure</div></div></blockquote><div><br class=""></div> Yes, I think it should be removed.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""><div class="">Barry: It seems we should either disable this option, or rename it to --with-cuda-arch and get it accepted by both kokkos.py and cuda.py</div><div class=""><br clear="all" class=""><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">--Junchao Zhang</div></div></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, May 16, 2021 at 6:15 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">progress:</div><div class=""><br class=""></div>04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check<br class="">Running check examples to verify correct installation<br class="">Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc<br class="">C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process<br class="">Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes<br class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br class="">srun: error: Unable to create step for job 1921749: More processors requested than permitted<br class="">2,15c2,32<br class="">< 0 SNES Function norm 2.391552133017e-01 <br class="">< 0 KSP Residual norm 2.325621076120e-01 <br class="">< 1 KSP Residual norm 1.654206318674e-02 <br class="">< 2 KSP Residual norm 7.202836119880e-04 <br class="">< 3 KSP Residual norm 1.796861424199e-05 <br class="">< 4 KSP Residual norm 2.461332992052e-07 <br class="">< 1 SNES Function norm 6.826585648929e-05 <br class="">< 0 KSP Residual norm 2.347339172985e-05 <br class="">< 1 KSP Residual norm 8.356798075993e-07 <br class="">< 2 KSP Residual norm 1.844045309619e-08 <br class="">< 3 KSP Residual norm 5.336386977405e-10 <br class="">< 4 KSP Residual norm 2.662608472862e-11 <br class="">< 2 SNES Function norm 6.549682264799e-11 <br class="">< Number of SNES iterations = 2<br class="">---<br class="">> Kokkos::Cuda::initialize ERROR: likely mismatch of architecture<br class="">> [cgpu11:39927] *** Process received signal ***<br class="">> [cgpu11:39927] Signal: Aborted (6)<br class="">> [cgpu11:39927] Signal code: (-6)<br class="">> [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360]<br class="">> [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160]<br class="">> [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741]<br class="">> [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5]<br class="">> [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c]<br class="">> [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224]<br class="">> [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1]<br class="">> [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809]<br class="">> [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b]<br class="">> [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224]<br class="">> [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596]<br class="">> [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94]<br class="">> [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f]<br class="">> [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a]<br class="">> [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30]<br class="">> [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f]<br class="">> [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d]<br class="">> [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a]<br class="">> [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5]<br class="">> [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de]<br class="">> [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1]<br class="">> [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69]<br class="">> [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a]<br class="">> [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba]<br class="">> [cgpu11:39927] *** End of error message ***<br class="">> srun: error: cgpu11: task 0: Aborted<br class="">> srun: Terminating job step 1921749.6<br class="">/global/homes/m/madams/petsc/src/snes/tutorials<br class="">Possible problem with ex19 running with cuda, diffs above<br class="">=========================================<br class="">gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)<br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, May 16, 2021 at 12:31 AM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.<div class=""><br class=""></div><div class=""> Mark,</div><div class=""><br class=""></div><div class=""> PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.</div><div class=""><br class=""></div><div class=""> If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">~/petsc</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)" class=""><b class=""> (main=)</b></span><span style="font-variant-ligatures:no-common-ligatures" class=""> arch-main</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets </span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">~/petsc</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(200,20,201)" class=""><b class=""> (main=)</b></span><span style="font-variant-ligatures:no-common-ligatures" class=""> arch-main</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o</span></div><div style="margin:0px;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 15, 2021, at 8:44 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">Add <span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap" class="">--show </span>to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options</div><div class=""><span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap" class=""><br class=""></span></div><div class="">PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/<a href="http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname" target="_blank" class="">3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname</a> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show<span style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;white-space:pre-wrap" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><div style="font-family:Menlo,Monaco,"Courier New",monospace;font-size:14px;line-height:21px;white-space:pre-wrap" class=""><div class=""><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;font-size:small" class="">--Junchao Zhang</span><br class=""></div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, May 15, 2021 at 7:14 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Thanks,<div class="">Now I get an error in make all. THis was working a few weeks ago.</div><div class="">Make.log was empty but here is the output.</div><div class=""> ....</div><div class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o<br class="">PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/<a href="http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname" target="_blank" class="">3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname</a> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o<br class="">g++: error: arch=compute_80: No such file or directory<br class="">g++: error: code=sm_80: No such file or directory<br class="">gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1<br class="">gmake[3]: *** Waiting for unfinished jobs....<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o<br class=""> CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o<br class=""> FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o<br class=""> CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o<br class=""> CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o<br class="">gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2<br class="">**************************ERROR*************************************<br class=""> Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log<br class=""> Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank" class="">petsc-maint@mcs.anl.gov</a><br class="">********************************************************************<br class="">gmake[1]: *** [makefile:40: all] Error 1<br class="">make: *** [GNUmakefile:9: all] Error 2<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, May 15, 2021 at 5:30 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0 <div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 15, 2021, at 2:51 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos.</div><div class="">Any suggestions?</div><div class="">Thanks,</div><div class="">Mark</div><div class=""><br class=""></div>============================================================================================= ============================================================================================= Trying to download git://<a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a> for SOWING ============================================================================================= ============================================================================================= Trying to download git://<a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a> for SOWING ============================================================================================= ============================================================================================= Trying to download <a href="https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz</a> for SOWING ============================================================================================= *******************************************************************************<br class=""> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):<br class="">-------------------------------------------------------------------------------<br class="">Error during download/extract/detection of SOWING:<br class="">Unable to clone sowing<br class="">Could not execute "['git clone <a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']":<br class="">Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'...<br class="">fatal: unable to access '<a href="https://bitbucket.org/petsc/pkg-sowing.git/" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git/</a>': The requested URL returned error: 503<br class="">Unable to download package SOWING from: git://<a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a><br class="">* If URL specified manually - perhaps there is a typo?<br class="">* If your network is disconnected - please reconnect and rerun ./configure<br class="">* Or perhaps you have a firewall blocking the download<br class="">* You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually<br class="">* or you can download the above URL manually, to /yourselectedlocation<br class=""> and use the configure option:<br class=""> --download-sowing=/yourselectedlocation<br class=""> Unable to clone sowing<br class="">Could not execute "['git clone <a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']":<br class="">Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'...<br class="">fatal: unable to access '<a href="https://bitbucket.org/petsc/pkg-sowing.git/" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git/</a>': The requested URL returned error: 503<br class="">Unable to download package SOWING from: git://<a href="https://bitbucket.org/petsc/pkg-sowing.git" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing.git</a><br class="">* If URL specified manually - perhaps there is a typo?<br class="">* If your network is disconnected - please reconnect and rerun ./configure<br class="">* Or perhaps you have a firewall blocking the download<br class="">* You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually<br class="">* or you can download the above URL manually, to /yourselectedlocation<br class=""> and use the configure option:<br class=""> --download-sowing=/yourselectedlocation<br class=""> file could not be opened successfully<br class="">Downloaded package SOWING from: <a href="https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz" target="_blank" class="">https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz</a> is not a tarball.<br class="">[or installed python cannot process compressed files]<br class="">* If you are behind a firewall - please fix your proxy and rerun ./configure<br class=""> For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to <a href="http://proxyout.lanl.gov/" target="_blank" class="">http://proxyout.lanl.gov</a><br class="">* You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually<br class="">* or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz<br class=""> and use the configure option:<br class=""> --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz<br class="">*******************************************************************************<br class=""></div>
<span id="gmail-m_2855113807534898539gmail-m_3718039331840808274gmail-m_1572418973608050119gmail-m_4467875874349699104cid:f_koq5z5j90" class=""><configure.log></span></div></blockquote></div><br class=""></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></body></html>