[petsc-users] snes/ex19 issue with nvprof

Tue Jul 2 21:40:01 CDT 2019

Hello everyone,

When I run the ex19 with cuda like this:
 mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor
-mat_type aijcusparse -vec_type cuda  -log_view

it worked fine and produced correct results.

However, when I tried to run this example with nvprof:
nvprof --profile-child-processes mpiexec -np 4 ./extest -da_refine 5
-snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda
 -log_view

I got errors like:
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: CUDA error 700
[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.11.2, unknown
[3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul  2 22:26:01
2019
[3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0
--with-mpi-dir=/home/eps/MyLocal/mpi/mpich
--with-blaslapack-dir=/home/eps/MyLocal/intel/mkl
--with-cuda-dir=/home/eps/MyLocal/cuda
--with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install
--download-hdf5=1
[3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in
/home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[3]PETSC ERROR: #2 VecSet() line 547 in
/home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c
[3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in
/home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
[3]PETSC ERROR: #4 VecSetType() line 51 in
/home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
[3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in
/home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
[3]PETSC ERROR: #6 VecSetType() line 51 in
/home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
[3]PETSC ERROR: #7 MatCreateVecs() line 8996 in
/home/eps/MyLocal/petsc/src/mat/interface/matrix.c
[3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in
/home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c
[3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in
/home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c
[3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in
/home/eps/MyLocal/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in
/home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c
[3]PETSC ERROR: #12 SNESSolve() line 4560 in
/home/eps/MyLocal/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c

The full run log is attached.

I am using  NVIDIA-SMI 418.56       Driver Version: 418.56       CUDA
Version: 10.1.

I do not know why it is okay without nvprof but crashed with nvprof. Any
suggestion to fix this?

Thank you.

Best,
Xiangdong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190702/911f7abb/attachment.html>
-------------- next part --------------
eps at new:~/MyCodes/petsctest$ nvprof --profile-child-processes mpiexec -np 4 ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda  -log_view
==4979== NVPROF is profiling process 4979, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4980== NVPROF is profiling process 4980, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4981== NVPROF is profiling process 4981, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4978== NVPROF is profiling process 4978, command: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1.
  0 SNES Function norm 1.036007954337e-02 
[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Error in external library
[3]PETSC ERROR: CUDA error 700
[3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[3]PETSC ERROR: Petsc Release Version 3.11.2, unknown 
[3]PETSC ERROR: ./ex19 on a arch-opt named new by eps Tue Jul  2 22:26:01 2019
[3]PETSC ERROR: Configure options PETSC_ARCH=arch-opt --with-debugging=0 --with-mpi-dir=/home/eps/MyLocal/mpi/mpich --with-blaslapack-dir=/home/eps/MyLocal/intel/mkl --with-cuda-dir=/home/eps/MyLocal/cuda --with-hypre-dir=/home/eps/MyLocal/hypre-2.15.1/hypre-install --download-hdf5=1
[3]PETSC ERROR: #1 VecSet_SeqCUDA() line 785 in /home/eps/MyLocal/petsc/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[3]PETSC ERROR: #2 VecSet() line 547 in /home/eps/MyLocal/petsc/src/vec/vec/interface/rvector.c
[3]PETSC ERROR: #3 VecCreate_MPICUDA() line 178 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
[3]PETSC ERROR: #4 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
[3]PETSC ERROR: #5 VecCreate_CUDA() line 192 in /home/eps/MyLocal/petsc/src/vec/vec/impls/mpi/mpicuda/mpicuda.cu
[3]PETSC ERROR: #6 VecSetType() line 51 in /home/eps/MyLocal/petsc/src/vec/vec/interface/vecreg.c
[3]PETSC ERROR: #7 MatCreateVecs() line 8996 in /home/eps/MyLocal/petsc/src/mat/interface/matrix.c
[3]PETSC ERROR: #8 MatFDColoringCreate() line 482 in /home/eps/MyLocal/petsc/src/mat/matfd/fdmatrix.c
[3]PETSC ERROR: #9 SNESComputeJacobian_DMDA() line 175 in /home/eps/MyLocal/petsc/src/snes/utils/dmdasnes.c
[3]PETSC ERROR: #10 SNESComputeJacobian() line 2718 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 222 in /home/eps/MyLocal/petsc/src/snes/impls/ls/ls.c
[3]PETSC ERROR: #12 SNESSolve() line 4560 in /home/eps/MyLocal/petsc/src/snes/interface/snes.c
[3]PETSC ERROR: #13 main() line 161 in /home/eps/MyCodes/petsctest/extest.c
[3]PETSC ERROR: PETSc Option Table entries:
[3]PETSC ERROR: -da_refine 5
[3]PETSC ERROR: -ksp_monitor
[3]PETSC ERROR: -log_view
[3]PETSC ERROR: -mat_type aijcusparse
[3]PETSC ERROR: -snes_monitor
[3]PETSC ERROR: -snes_view
[3]PETSC ERROR: -vec_type cuda
[3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 3
==4981== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4981== Profiling result:
No kernels were profiled.
No API activities were profiled.
==4981== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
==4979== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4979== Profiling result:
No kernels were profiled.
No API activities were profiled.
==4979== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
==4980== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4980== Profiling result:
No kernels were profiled.
No API activities were profiled.
==4980== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
==4978== Profiling application: ./ex19 -da_refine 5 -snes_view -snes_monitor -ksp_monitor -mat_type aijcusparse -vec_type cuda -log_view
==4978== Profiling result:
No kernels were profiled.
No API activities were profiled.
==4978== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
======== Error: Application returned non-zero code 76