[petsc-dev] [petsc-users] VECMPICUSP with ghosted vector

[Barry Smith]

   Fredrik,

    This question belongs on petsc-dev at mcs.anl.gov since it involves additions/extensions to PETSc so I am moving the discussion over to there.

    We have not done the required work to have ghosted vectors work with CUSP yet, so this will require some additions to PETSc. We can help you with that process but since the PETSc team does not have a CUSP person developing PETSc full time you will need to actual contribute some code but I'll try to guide you in the right direction.

     The first observation is that ghosted vectors in PETSc are actually handled with largely the same code as VECMPI vectors (with just no ghost points by default) so in theory little work needs to be done to get the functionality you need. What makes the needed changes non-trivial is the current interface where one calls VecCreateGhost() to create the vectors.   This is one of our "easy" interfaces and it is somewhat legacy in that there is no way to control the types of the vectors since it creates everything about the vector in one step.   Note that we have the same issues with regard to the pthread versions of the PETSc vectors and ghosting.

    So before we even talk about what code to change/add we need to decide on the interface.  Presumably you want to be able to decide at runtime whether to use regular VECMPI, VECMPICUSP or VECMPIPTHREAD  in your ghosted vectors. How do we get that information in there? An additional argument to VecCreateGhost() (ugly?)? Options database (by calling VecSetFromOptions() ?), other ways?  So for example one could have:

VecCreateGhost(......)
VecSetFromOptions(......)

to set the specific type cusp or pthread? What about 

VecCreateGhost(......)
VecSetType(......,VECMPICUSP);   

which as you note doesn't currently work. Note that the PTHREAD version needs to do its own memory allocation so essentially has to undo much of what VecCreateGhost() already did, is that a bad thing?

Or do we get rid of VecCreateGhost() completely and change the model to something like

VecCreate()
VecSetType()
VecSetGhosted()

or 

VecCreate()
VecSetTypeFromOptions()
VecSetGhosted()

or even 

VecCreate()
VecSetGhosted()   which will just default to regular MPI ghosted.

this model allows a clean implementation that doesn't require undoing previously built internals.


Everyone chime in with observations so we can figure out any refactorizations needed.

   Barry





On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote:

> Hi,
> 
> In FEniCS, we use ghosted vectors for parallel computations, with the functions
> 
> VecCreateGhost
> VecGhostGetLocalForm
> 
> As I am integrating the new GPU-based vectors and matrices in FEniCS, I want the ghosted vectors to be of type VECMPICUSP. I have tried to do this by calling VecSetType after creating the vector, but that makes VecGhostGetLocalForm give an error. 
> 
> Is it possible to set the type to be mpicusp when using ghost vectors, without changing much of the code? If so, how?
> 
> If not, how would you recommend I proceed to work with mpicusp vectors in this context?
> 
> Thanks!
> 
> -- 
> Fredrik