[petsc-users] cudaSetDevice

[Harshad Sahasrabudhe]

Hi Jed,

You are right, it doesn't if you use just one PETSc. In our case, we use 2 PETSc libraries (compiled separately, one each for complex and double). And in that case, when the second PETSc library is initialized, cudaSetDevice throws an error. I can't think of any way in which both the PETSc libraries could be initialized with CUDA on the same process.

Do you have any idea?

I can only think of turning CUDA off on one PETSc as a way to avoid this.

Harshad

----- Original Message -----
From: "Jed Brown" <jedbrown at mcs.anl.gov>
To: "Harshad Sahasrabudhe" <hsahasra at purdue.edu>, "Matthew Knepley" <knepley at gmail.com>
Cc: petsc-users at mcs.anl.gov
Sent: Thursday, September 26, 2013 8:37:36 PM
Subject: Re: [petsc-users] cudaSetDevice

Harshad Sahasrabudhe <hsahasra at purdue.edu> writes:

> Basically, PetscInitialize is called 2 times, the second time from
> libmesh. 

It should not be.

$ git grep -C3 PetscInitialize src/base/libmesh.C
src/base/libmesh.C-      // Check whether the calling program has already initialized
src/base/libmesh.C-      // PETSc, and avoid duplicate Initialize/Finalize
src/base/libmesh.C-      PetscBool petsc_already_initialized;
src/base/libmesh.C:      ierr = PetscInitialized(&petsc_already_initialized);
src/base/libmesh.C-             CHKERRABORT(libMesh::COMM_WORLD,ierr);
src/base/libmesh.C-      if (petsc_already_initialized != PETSC_TRUE)
src/base/libmesh.C-        libmesh_initialized_petsc = true;
--
src/base/libmesh.C-# else
src/base/libmesh.C-      if (libmesh_initialized_petsc)
src/base/libmesh.C-        {
src/base/libmesh.C:          ierr = PetscInitialize (&argc, const_cast<char***>(&argv), NULL, NULL);
src/base/libmesh.C-              CHKERRABORT(libMesh::COMM_WORLD,ierr);
src/base/libmesh.C-        }
src/base/libmesh.C-# endif