[petsc-users] ISLocalToGlobalMapping + VecCreateGhost

[Mohammad Mirzadeh]

Ok I think I get it now. I'll look into DM (and probably come back
with more questions!)

Thanks

On Wed, Oct 2, 2013 at 11:36 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Oct 2, 2013 at 1:32 PM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
>>
>> Matt,
>>
>> I just discovered that my approach does not work properly with
>> VecGhostUpdateBegin/End -- I get wrong data after ghost update
>> process. Any idea why? When I change the IS, is PETSc aware that my
>> ghost values are not just at the end of vector?
>
>
> Then I was probably misunderstanding what you want to do.
>
>   1) L2GM is for translating local to global indices automatically, but
> knows nothing about "ghosting" in the Vec
>
>   2) For ghosting, you have a local vector and global vector and a
> VecScatter that maps between them
>
>   3) VecGhost is a special kind of 2) since we know that the local vector
> fits in side the global vec
>
>   4) Since you break relationship 3), I would just use the DM (as I say
> below).
>
>   Matt
>
>>
>> Mohammad
>>
>> On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
>> > On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
>> > wrote:
>> >>
>> >> Hi guys,
>> >>
>> >> I just did something by pure guessing which seems to work and I want
>> >> to make sure its the right thing!
>> >>
>> >> I have a specific layout for my vectors that look like this
>> >>
>> >>  --------------------------------------------------------------
>> >> | ghost values | local values | ghost values |
>> >>  --------------------------------------------------------------
>> >> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
>> >>
>> >> which means all nodes with index [0,m) and [m+n, N) are to be treated
>> >> as ghost and all intermediate ones as local. Since PETSc stores the
>> >> ghost values at the end of ghosted vec, so far I have been manually
>> >> mapping back and forth between petsc and my application numbering.
>> >> After profiling my code, it turned out that about 15% of run-time was
>> >> just doing this mapping which is absolutely ridiculous!
>> >>
>> >> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
>> >> using my own application original local2global relation shown above.
>> >> Then I create the petsc vec like this:
>> >>
>> >> // Create PetscVec
>> >> // We do not care about the actual global index of ghost nodes at this
>> >> point
>> >> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
>> >> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
>> >> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
>> >>
>> >>
>> >> // Apply mapping
>> >> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
>> >>
>> >> After this point I do the usual VecGetArray on the vec and set the
>> >> values, but this time without any extra mapping ... my very simple
>> >> test seems to be ok. Is this the correct way of using
>> >> ISLocalToGlobalMapping?
>> >
>> >
>> > Yes, this is the intention. And at a higher level, PETSc now tends to
>> > use a
>> > DM to
>> > organize this process. You tell the DM about your data layout (somewhat
>> > like
>> > giving
>> > the L2G mapping) and then you can DMGetLocalVector(),
>> > DMGetGlobalVector(),
>> > and
>> > DMLocalToGlobal().
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> >>
>> >> I guess I'm suspicious because VecCreateGhost is supposed to
>> >> internally set the mapping and it is supposed to position ghost nodes
>> >> at the end of the array which I don't want it to do...
>> >>
>> >> Thanks and sorry about the long email!
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> > experiments
>> > is infinitely more interesting than any results to which their
>> > experiments
>> > lead.
>> > -- Norbert Wiener
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener