[petsc-users] snes/ex19 issue with nvprof

[Matthew Knepley]

On Tue, Jul 2, 2019 at 9:41 PM Xiangdong via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hello everyone,
>
> When I run the ex19 with cuda like this:
>  mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor
> -mat_type aijcusparse -vec_type cuda  -log_view
>
> it worked fine and produced correct results.
>
> However, when I tried to run this example with nvprof:
> nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5
> -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda
>  -log_view
>
> I got errors like:
> [3]PETSC ERROR: Error in external library
> [3]PETSC ERROR: CUDA error 700
>

Looking at Google for this error, it seems that it tends to arise with
incompatible drivers. Perhaps your nvprof is
older/newer than the library PETSc is linked against?

   Matt


> [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [3]PETSC ERROR: Petsc Release Version 3.11.2, unknown
> [3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul  2 22:26:01
> 2019
> [3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0
> --with-mpi-dir=/home/eps/MyLocal/mpi/mpich
> --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl
> --with-cuda-dir=/home/eps/MyLocal/cuda
> --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install
> --download-hdf5=1
> [3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in
> /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [3]PETSC ERROR: #2 VecSet() line 547 in
> /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c
> [3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in
> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
> [3]PETSC ERROR: #4 VecSetType() line 51 in
> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
> [3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in
> /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
> [3]PETSC ERROR: #6 VecSetType() line 51 in
> /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
> [3]PETSC ERROR: #7 MatCreateVecs() line 8996 in
> /home/eps/MyLocal/petsc/src/mat/interface/matrix.c
> [3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in
> /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c
> [3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in
> /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c
> [3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in
> /home/eps/MyLocal/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in
> /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c
> [3]PETSC ERROR: #12 SNESSolve() line 4560 in
> /home/eps/MyLocal/petsc/src/snes/interface/snes.c
> [3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c
>
> The full run log is attached.
>
> I am using  NVIDIA-SMI 418.56       Driver Version: 418.56       CUDA
> Version: 10.1.
>
> I do not know why it is okay without nvprof but crashed with nvprof. Any
> suggestion to fix this?
>
> Thank you.
>
> Best,
> Xiangdong
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190703/387e2d9a/attachment.html>