[petsc-users] cudaSetDevice

[Matthew Knepley]

On Thu, Sep 26, 2013 at 3:40 PM, Harshad Sahasrabudhe
<hsahasra at purdue.edu>wrote:

> Hi,
>
> In our code, PetscInitialize() is called initially. Then, depending on
> whether we need libmesh or not, LibMeshInit() is called. This works fine
> when we are not using CUDA in PETSc. But when we are using CUDA,
> cudaSetDevice() gets called twice which throws an error:
>

So you are saying cudaSetDevice() gets called before PetscInit? What would
you propose for us? The problem is
that we have no abstraction (like the communicator) which can tell us that
you have called it.

   Matt


> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Error in system call!
> [0]PETSC ERROR: error in cudaSetDevice cannot set while device is active
> in this process!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development GIT revision:   GIT Date:
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab
> Thu Sep 26 17:26:43 2013
> [0]PETSC ERROR: Libraries linked from
> /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib
> [0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013
> [0]PETSC ERROR: Configure options --with-x=0 --with-hdf5
> --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48
> --with-scalar-type=real --with-single-library=0 --with-pic=1
> --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++
> --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn "
> --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3
> --download-metis=1 --download-parmetis=1 --download-superlu_dist=1
> --download-mumps=1 --download-scalapack=1 --download-blacs=1
> --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35
> --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master
> --with-cuda=1 --with-thrust=1
> --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in
> /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c
> [0]PETSC ERROR: PetscInitialize() line 788 in
> /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c
> [0]PETSC ERROR: LibMeshInit() line 457 in
> "unknowndirectory/"src/base/libmesh.C
>
>
> Can we tell PETSc when to run cudaSetDevice()? Or is there any other way
> to avoid this?
>
> Thanks,
> Harshad
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20130926/26262a4b/attachment.html>