<div dir="ltr">Thanks Rich,<div>I'm not sure what was wrong but I redid everything from scratch and it works.</div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 2, 2019 at 4:26 PM Mills, Richard Tran via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">



<div bgcolor="#FFFFFF">
Mark,<br>
<br>
It looks like you are missing some critical CUDA library (or libraries) in your link line. I know you will at least need the CUDA runtime "-lcudart". Look at something like PETSC_WITH_EXTERNAL_LIB for one of your CUDA-enabled PETSc builds in $PETSC_ARCH/lib/petsc/conf/petscvariables
 to see what else you might need.<br>
<br>
--Richard<br>
<br>
<div>On 10/2/19 7:20 AM, Mark Adams via petsc-dev wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I found a CUDAVersion.cu of STREAMS and tried to build it. I got it to compile manually with:</div>
<div><br>
</div>
<div>
<div>nvcc -o CUDAVersion.o -ccbin pgc++ -I/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/pgi-19.4/spectrum-mpi-10.3.0.1-20190611-4ymaahbai7ehhw4rves5jjiwon2laz3a/include -Wno-deprecated-gpu-targets -c --compiler-options="-g
 -I/ccs/home/adams/petsc/include -I/ccs/home/adams/petsc/arch-summit-opt64-pgi-cuda/include   " `pwd`/CUDAVersion.cu<br>
/gpfs/alpine/geo127/scratch/adams/CUDAVersion.cu(22): warning: conversion from a string literal to "char *" is deprecated<br>
</div>
<div> ....</div>
<div><br>
</div>
<div>And this did produce a .o file. But I get this when I try to link.</div>
<div><br>
</div>
<div>make -f makestreams CUDAVersion<br>
mpicc -g -fast  -o CUDAVersion CUDAVersion.o -Wl,-rpath,/ccs/home/adams/petsc/arch-summit-opt64-pgi-cuda/lib -L/ccs/home/adams/petsc/arch-summit-opt64-pgi-cuda/lib -Wl,-rpath,/ccs/home/adams/petsc/arch-summit-opt64-pgi-cuda/lib -L/ccs/home/adams/petsc/arch-summit-opt64-pgi-cuda/lib
 /autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/pgi-19.4-6acz4xyqjlpoaonjiiqjme2aknrfnzoy/linuxpower/19.4/lib/pgi.ld -Wl,-rpath,/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/pgi-19.4/spectrum-mpi-10.3.0.1-20190611-4ymaahbai7ehhw4rves5jjiwon2laz3a/lib
 -L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/pgi-19.4/spectrum-mpi-10.3.0.1-20190611-4ymaahbai7ehhw4rves5jjiwon2laz3a/lib -Wl,-rpath,/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/pgi-19.4-6acz4xyqjlpoaonjiiqjme2aknrfnzoy/linuxpower/19.4/lib
 -L/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/pgi-19.4-6acz4xyqjlpoaonjiiqjme2aknrfnzoy/linuxpower/19.4/lib -Wl,-rpath,/usr/lib/gcc/ppc64le-redhat-linux/4.8.5 -L/usr/lib/gcc/ppc64le-redhat-linux/4.8.5 -lpetsc -llapack
 -lblas -lparmetis -lmetis -lstdc++ -ldl -lpthread -lmpiprofilesupport -lmpi_ibm_usempi -lmpi_ibm_mpifh -lmpi_ibm -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -latomic -lpgkomp -lomp -lomptarget -lpgmath -lpgc -lrt -lmass_simdp9 -lmassvp9 -lmassp9 -lm
 -lgcc_s -lstdc++ -ldl<br>
CUDAVersion.o: In function `setupStream(long, PetscBool, PetscBool)':<br>
/gpfs/alpine/geo127/scratch/adams/CUDAVersion.cu:394: undefined reference to `cudaGetDeviceCount'<br>
/gpfs/alpine/geo127/scratch/adams/CUDAVersion.cu:406: undefined reference to `cudaSetDevice'<br>
</div>
<div> ....</div>
<div><br>
</div>
<div>I have compared this link line with working examples and it looks the same. There is not .c file here -- main is in the .cu file. I assume that is the difference.</div>
<div><br>
</div>
<div>Any ideas?</div>
<div>Thanks,</div>
<div>Mark</div>
</div>
</div>
</blockquote>
<br>
</div>

</blockquote></div>