<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> configure prints the information about CUDA at the end of the run, you can check that information to see which was actually used. <div class=""><br class=""></div><div class=""> I have a new MR where PETSc records the gencodearch it was built with and then when your program starts up CUDA it verifies that the hardware supports the gencodearch it was built with. Hopefully this will alleviate difficulties in the future. Of course this won't help when using libraries that use CUDA built externally from PETSc.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On May 18, 2021, at 10:30 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><b class=""> '--with-cuda-gencodearch=70',</b><div class=""><br clear="all" class=""><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 18, 2021 at 6:29 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Damn, I am getting this problem on Summit and did a clean configure. <div class="">I removed the Kokkos arch=70 line and added </div><div class=""> '--with-cudac-gencodearch=70',<br class=""><div class=""><br class=""></div><div class="">Any ideas?</div><div class=""><br class=""></div><div class="">< Number of SNES iterations = 2<br class="">---<br class="">> Kokkos::Cuda::initialize ERROR: likely mismatch of architecture<br class="">> [h50n11:35759] *** Process received signal ***<br class="">> [h50n11:35759] Signal: Aborted (6)<br class="">> [h50n11:35759] Signal code: (-6)<br class="">> [h50n11:35759] [ 0] [0x2000000504d8]<br class="">> [h50n11:35759] [ 1] /lib64/libc.so.6(abort+0x2b4)[0x200032322094]<br class="">> [h50n11:35759] [ 2] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl10host_abortEPKc+0x58)[0x20000f944558]<br class="">> [h50n11:35759] [ 3] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xe60)[0x20000f95c210]<br class="">> [h50n11:35759] [ 4] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Cuda15impl_initializeENS0_12SelectDeviceEm+0x30)[0x20000f95c2b0]<br class="">> [h50n11:35759] [ 5] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl20CudaSpaceInitializer10initializeERKNS_13InitArgumentsE+0x34)[0x20000f95c314]<br class="">> [h50n11:35759] [ 6] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x60)[0x20000f926aa0]<br class="">> [h50n11:35759] [ 7] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_backendsERKNS_13InitArgumentsE+0x2c)[0x20000f926dac]<br class="">> [h50n11:35759] [ 8] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x2c)[0x20000f92b73c]<br class="">> [h50n11:35759] [ 9] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos10initializeENS_13InitArgumentsE+0x2c)[0x20000f92d08c]<br class="">> [h50n11:35759] [10] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f4)[0x200000343424]<br class="">> [h50n11:35759] [11] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x68b3dc)[0x20000077b3dc]<br class="">> [h50n11:35759] [12] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x3119b4)[0x2000004019b4]<br class="">> [h50n11:35759] [13] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x306790)[0x2000003f6790]<br class="">> [h50n11:35759] [14] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x274d24)[0x200000364d24]<br class="">> [h50n11:35759] [15] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0xd4)[0x200000412504]<br class="">> [h50n11:35759] [16] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x329f9c)[0x200000419f9c]<br class="">> [h50n11:35759] [17] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(VecScatterBegin+0x9c)[0x20000041fa8c]<br class="">> [h50n11:35759] [18] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x30)[0x2000010ef560]<br class="">> [h50n11:35759] [19] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x290)[0x2000013314b0]<br class="">> [h50n11:35759] [20] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x1513c10)[0x200001603c10]<br class="">> [h50n11:35759] [21] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESComputeFunction+0x164)[0x200001625584]<br class="">> [h50n11:35759] [22] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x15830f4)[0x2000016730f4]<br class="">> [h50n11:35759] [23] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESSolve+0x814)[0x200001634c44]<br class="">> [h50n11:35759] [24] ./ex19[0x10001a70]<br class="">> [h50n11:35759] [25] /lib64/libc.so.6(+0x25200)[0x200032305200]<br class="">> [h50n11:35759] [26] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000323053f4]<br class="">> [h50n11:35759] *** End of error message ***<br class="">> ERROR: One or more process (first noticed rank 0) terminated with signal 6 (core dumped)<br class="">/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials<br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 17, 2021 at 8:24 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I thought I did a clean make but I made a clean one now and it seems to be working now.<div class=""><br class=""><div class="">Also, I am trying to fix this error message that I get on Cori with 'make check'.</div><div class="">I set mpiexec='srun -G 2 -c 20' and get an interactive shell with these parameters, but I get error messages on Kokkos:</div><div class=""><br class=""></div><div class="">Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes<br class="">See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank" class="">http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br class=""><b class="">srun: error: Unable to create step for job 1923618: More processors requested than permitted<br class=""></b>C/C++ example src/snes/tutorials/ex19 run successfully with cuda<br class="">gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)<br class="">1,25c1<br class="">< atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000<br class="">< Vec Object: Exact Solution 2 MPI processes<br class="">< type: mpikokkos<br class="">< Process [0]<br class="">< 0.<br class="">< 0.015625<br class="">< 0.125<br class="">< Process [1]<br class="">< 0.421875<br class="">< 1.<br class="">< Vec Object: Forcing function 2 MPI processes<br class="">< type: mpikokkos<br class="">< Process [0]<br class="">< 1e-72<br class="">< 1.50024<br class="">< 3.01563<br class="">< Process [1]<br class="">< 4.67798<br class="">< 7.<br class="">< 0 SNES Function norm 5.414682427127e+00 <br class="">< 1 SNES Function norm 2.952582418265e-01 <br class="">< 2 SNES Function norm 4.502293658739e-04 <br class="">< 3 SNES Function norm 1.389665806646e-09 <br class="">< Number of SNES iterations = 3<br class="">< Norm of error 1.49752e-10 Iterations 3<br class="">---<br class=""><b class="">> srun: error: Unable to create step for job 1923618: More processors requested than permitted<br class=""></b>/global/homes/m/madams/petsc/src/snes/tutorials<br class="">Possible problem with ex3k running with kokkos-kernels, diffs above<br class="">=========================================<br class="">Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process<br class="">Completed test examples<br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, May 16, 2021 at 11:14 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class="">Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to?</div><div class=""><br class=""></div><div class="">Sorry, but this may require another rm -rf arch* and running ./configure again.</div><div class=""><br class=""></div><div class=""><a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c2178246db0a94a430e0038b6af535e7e53d3f21e2437e8977b8c2e" target="_blank" class="">https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c2178246db0a94a430e0038b6af535e7e53d3f21e2437e8977b8c2e</a></div><div class=""><br class=""></div><div class=""><br class=""></div><dl class=""><dt style="font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class=""><span style="display:block;background-color:rgb(239,239,240);border-top:1px solid rgb(216,220,222);border-bottom:1px solid rgb(238,243,245);padding:3px" class=""><span style="background-color:yellow" class="">cudaErrorInvalidDeviceFunction</span> = <span class="">98</span></span></dt><dd style="margin-left:50px;margin-bottom:5px;margin-top:2px;font-family:"Trebuchet MS","DIN Pro",sans-serif;font-size:14px" class="">The requested device function does not exist or is not compiled for the proper device architecture.</dd></dl><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On May 16, 2021, at 9:09 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">I now get this error. A blas error from VecAXPBYPCZ ...</div><div dir="ltr" class="">Any ideas?<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">terminate called after throwing an instance of 'std::runtime_error'<br class=""> what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654<br class="">Traceback functionality not available<br class=""><br class="">[cgpu16:55192] *** Process received signal ***<br class="">[cgpu16:55192] Signal: Aborted (6)<br class="">[cgpu16:55192] Signal code: (-6)<br class="">[cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360]<br class="">[cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160]<br class="">[cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741]<br class="">[cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83]<br class="">[cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6]<br class="">[cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21]<br class="">[cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053]<br class="">[cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f]<br class="">[cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d]<br class="">[cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7]<br class="">[cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1]<br class="">[cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781]<br class="">[cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b]<br class="">[cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1]<br class="">[cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e]<br class="">[cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a]<br class="">[cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675]<br class="">[cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e]<br class="">[cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651]<br class="">[cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c]<br class="">[cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05]<br class="">[cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455]<br class="">[cgpu16:55192] [22] ../ex2-kok[0x4033eb]<br class="">[cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a]<br class="">[cgpu16:55192] [24] ../ex2-kok[0x404aaa]<br class="">[cgpu16:55192] *** End of error message ***<br class="">/global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@"<br class="">0 stopping nvidia-cuda-mps-control on cgpu16</div></div></div>
</div></blockquote></div><br class=""></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>