[petsc-dev] [petsc-users] VECMPICUSP with ghosted vector

[Matthew Knepley]

On Mon, Feb 6, 2012 at 11:09 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>
>   Fredrik,
>
>    This question belongs on petsc-dev at mcs.anl.gov since it involves
> additions/extensions to PETSc so I am moving the discussion over to there.
>
>    We have not done the required work to have ghosted vectors work with
> CUSP yet, so this will require some additions to PETSc. We can help you
> with that process but since the PETSc team does not have a CUSP person
> developing PETSc full time you will need to actual contribute some code but
> I'll try to guide you in the right direction.
>
>     The first observation is that ghosted vectors in PETSc are actually
> handled with largely the same code as VECMPI vectors (with just no ghost
> points by default) so in theory little work needs to be done to get the
> functionality you need. What makes the needed changes non-trivial is the
> current interface where one calls VecCreateGhost() to create the vectors.
> This is one of our "easy" interfaces and it is somewhat legacy in that
> there is no way to control the types of the vectors since it creates
> everything about the vector in one step.   Note that we have the same
> issues with regard to the pthread versions of the PETSc vectors and
> ghosting.
>
>    So before we even talk about what code to change/add we need to decide
> on the interface.  Presumably you want to be able to decide at runtime
> whether to use regular VECMPI, VECMPICUSP or VECMPIPTHREAD  in your ghosted
> vectors. How do we get that information in there? An additional argument to
> VecCreateGhost() (ugly?)? Options database (by calling VecSetFromOptions()
> ?), other ways?  So for example one could have:
>
> VecCreateGhost(......)
> VecSetFromOptions(......)
>
> to set the specific type cusp or pthread? What about
>
> VecCreateGhost(......)
> VecSetType(......,VECMPICUSP);
>
> which as you note doesn't currently work. Note that the PTHREAD version
> needs to do its own memory allocation so essentially has to undo much of
> what VecCreateGhost() already did, is that a bad thing?
>
> Or do we get rid of VecCreateGhost() completely and change the model to
> something like
>
> VecCreate()
> VecSetType()
> VecSetGhosted()
>
> or
>
> VecCreate()
> VecSetTypeFromOptions()
> VecSetGhosted()
>
> or even
>
> VecCreate()
> VecSetGhosted()   which will just default to regular MPI ghosted.
>
> this model allows a clean implementation that doesn't require undoing
> previously built internals.
>

I am for the second model, just absorbing ghosting into the current
implementations.

    Matt


>
> Everyone chime in with observations so we can figure out any
> refactorizations needed.
>
>   Barry
>
>
>
>
>
> On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote:
>
> > Hi,
> >
> > In FEniCS, we use ghosted vectors for parallel computations, with the
> functions
> >
> > VecCreateGhost
> > VecGhostGetLocalForm
> >
> > As I am integrating the new GPU-based vectors and matrices in FEniCS, I
> want the ghosted vectors to be of type VECMPICUSP. I have tried to do this
> by calling VecSetType after creating the vector, but that makes
> VecGhostGetLocalForm give an error.
> >
> > Is it possible to set the type to be mpicusp when using ghost vectors,
> without changing much of the code? If so, how?
> >
> > If not, how would you recommend I proceed to work with mpicusp vectors
> in this context?
> >
> > Thanks!
> >
> > --
> > Fredrik
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120206/13180f67/attachment.html>