[petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell

[Barry Smith]

   Jed,

    The issue is not allowing or even often using SF. The issue is that DMGlobalToLocal() already has the flexibility to use a globaltolocal function pointer. And I object to hiding that or pretending it doesn't exist and saying one HAS to use SF especially if one already has code that does exactly what is needed some other way.


    Barry

On Feb 21, 2013, at 1:45 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> 
> On Thu, Feb 21, 2013 at 1:23 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Thu, Feb 21, 2013 at 1:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
> Yes, he needs the local to global scatters. Rich, I think the easiest way to do this now may be
> to define your data layout as a PetscSection (these are all local) using DMSetDefaultSection(),
> and record who owns which point using a PetscSF with DMSetPointSF().
> 
> Using this information, the DM will automatically construct a scatter between local and global,
> and the CreateLocal/GlobalVector() functions will also work right. This is what its is designed for.
> 
> Does this make sense?
> 
> 
> Rich, this will work fine if you don't mind having PETSc do the communication. It will be done using PetscSFBcastBegin/End, which should have similar performance to VecScatterBegin/End. (I should do a performance test of that.) If you want to use your own internal communication routines, then we need to add the DMShell functions to get your functions called.
> 
> It occurs to me that the description above might be pretty confusing. The idea is that we start with your local space and lump it together into (possibly variable-size) chunks, usually one per topological point in the mesh. Matt would describe this chunking with a PetscSection where the chunks are indexed by "point" (an integer).
> 
> The next component is that we have to describe how the points fit together, so we make a PetscSF where the "roots" are the global points, referred to by (rank, offset) pairs or global indices, and the "leaves" are the local space. You use PetscSFSetGraph() to create this, which Matt calls the "Point SF".
> 
> If you tell the DM about these two things by using DMSetDefaultSection() and DMSetPointSF(), then DM will create the SF to communicate the Vec data (using simple composition). You could also just use DMSetDefaultSF() directly, in which case you don't ever need to touch a PetscSection.
> 
> 
> The alternative to use VecScatter is also fine, but sometimes it is easier to use PetscSF. The Point SF, in particular, is very useful for handling integer data in the setup phase. Its API is richer than VecScatter, in addition to supporting more data types.