[petsc-users] ISLocalToGlobalMapping + VecCreateGhost

[Matthew Knepley]

On Wed, Oct 2, 2013 at 1:32 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>wrote:

> Matt,
>
> I just discovered that my approach does not work properly with
> VecGhostUpdateBegin/End -- I get wrong data after ghost update
> process. Any idea why? When I change the IS, is PETSc aware that my
> ghost values are not just at the end of vector?
>

Then I was probably misunderstanding what you want to do.

  1) L2GM is for translating local to global indices automatically, but
knows nothing about "ghosting" in the Vec

  2) For ghosting, you have a local vector and global vector and a
VecScatter that maps between them

  3) VecGhost is a special kind of 2) since we know that the local vector
fits in side the global vec

  4) Since you break relationship 3), I would just use the DM (as I say
below).

  Matt


> Mohammad
>
> On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> > wrote:
> >>
> >> Hi guys,
> >>
> >> I just did something by pure guessing which seems to work and I want
> >> to make sure its the right thing!
> >>
> >> I have a specific layout for my vectors that look like this
> >>
> >>  --------------------------------------------------------------
> >> | ghost values | local values | ghost values |
> >>  --------------------------------------------------------------
> >> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
> >>
> >> which means all nodes with index [0,m) and [m+n, N) are to be treated
> >> as ghost and all intermediate ones as local. Since PETSc stores the
> >> ghost values at the end of ghosted vec, so far I have been manually
> >> mapping back and forth between petsc and my application numbering.
> >> After profiling my code, it turned out that about 15% of run-time was
> >> just doing this mapping which is absolutely ridiculous!
> >>
> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
> >> using my own application original local2global relation shown above.
> >> Then I create the petsc vec like this:
> >>
> >> // Create PetscVec
> >> // We do not care about the actual global index of ghost nodes at this
> >> point
> >> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
> >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
> >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
> >>
> >>
> >> // Apply mapping
> >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
> >>
> >> After this point I do the usual VecGetArray on the vec and set the
> >> values, but this time without any extra mapping ... my very simple
> >> test seems to be ok. Is this the correct way of using
> >> ISLocalToGlobalMapping?
> >
> >
> > Yes, this is the intention. And at a higher level, PETSc now tends to
> use a
> > DM to
> > organize this process. You tell the DM about your data layout (somewhat
> like
> > giving
> > the L2G mapping) and then you can DMGetLocalVector(),
> DMGetGlobalVector(),
> > and
> > DMLocalToGlobal().
> >
> >   Thanks,
> >
> >      Matt
> >
> >>
> >> I guess I'm suspicious because VecCreateGhost is supposed to
> >> internally set the mapping and it is supposed to position ghost nodes
> >> at the end of the array which I don't want it to do...
> >>
> >> Thanks and sorry about the long email!
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments
> > is infinitely more interesting than any results to which their
> experiments
> > lead.
> > -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20131002/3b39996d/attachment.html>