[petsc-users] cudaSetDevice

[Harshad Sahasrabudhe]

Hi,

In our code, PetscInitialize() is called initially. Then, depending on whether we need libmesh or not, LibMeshInit() is called. This works fine when we are not using CUDA in PETSc. But when we are using CUDA, cudaSetDevice() gets called twice which throws an error:


[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Error in system call!
[0]PETSC ERROR: error in cudaSetDevice cannot set while device is active in this process!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Development GIT revision:   GIT Date: 
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab Thu Sep 26 17:26:43 2013
[0]PETSC ERROR: Libraries linked from /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib
[0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013
[0]PETSC ERROR: Configure options --with-x=0 --with-hdf5 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48 --with-scalar-type=real --with-single-library=0 --with-pic=1 --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++ --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn " --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35 --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master --with-cuda=1 --with-thrust=1 --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c
[0]PETSC ERROR: PetscInitialize() line 788 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c
[0]PETSC ERROR: LibMeshInit() line 457 in "unknowndirectory/"src/base/libmesh.C


Can we tell PETSc when to run cudaSetDevice()? Or is there any other way to avoid this?

Thanks,
Harshad