[petsc-users] ISLocalToGlobalMapping + VecCreateGhost

[Mohammad Mirzadeh]

Matt,

I just discovered that my approach does not work properly with
VecGhostUpdateBegin/End -- I get wrong data after ghost update
process. Any idea why? When I change the IS, is PETSc aware that my
ghost values are not just at the end of vector?

Mohammad

On Wed, Oct 2, 2013 at 4:13 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Oct 2, 2013 at 12:27 AM, Mohammad Mirzadeh <mirzadeh at gmail.com>
> wrote:
>>
>> Hi guys,
>>
>> I just did something by pure guessing which seems to work and I want
>> to make sure its the right thing!
>>
>> I have a specific layout for my vectors that look like this
>>
>>  --------------------------------------------------------------
>> | ghost values | local values | ghost values |
>>  --------------------------------------------------------------
>> 0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N
>>
>> which means all nodes with index [0,m) and [m+n, N) are to be treated
>> as ghost and all intermediate ones as local. Since PETSc stores the
>> ghost values at the end of ghosted vec, so far I have been manually
>> mapping back and forth between petsc and my application numbering.
>> After profiling my code, it turned out that about 15% of run-time was
>> just doing this mapping which is absolutely ridiculous!
>>
>> Anyway, to fix this now I set up an ISLocalToGlobalMapping object
>> using my own application original local2global relation shown above.
>> Then I create the petsc vec like this:
>>
>> // Create PetscVec
>> // We do not care about the actual global index of ghost nodes at this
>> point
>> std::vector<PetscInt> ghost_nodes(N - n -1, 0);
>> ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
>> (const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);
>>
>>
>> // Apply mapping
>> ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);
>>
>> After this point I do the usual VecGetArray on the vec and set the
>> values, but this time without any extra mapping ... my very simple
>> test seems to be ok. Is this the correct way of using
>> ISLocalToGlobalMapping?
>
>
> Yes, this is the intention. And at a higher level, PETSc now tends to use a
> DM to
> organize this process. You tell the DM about your data layout (somewhat like
> giving
> the L2G mapping) and then you can DMGetLocalVector(), DMGetGlobalVector(),
> and
> DMLocalToGlobal().
>
>   Thanks,
>
>      Matt
>
>>
>> I guess I'm suspicious because VecCreateGhost is supposed to
>> internally set the mapping and it is supposed to position ghost nodes
>> at the end of the array which I don't want it to do...
>>
>> Thanks and sorry about the long email!
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments
> is infinitely more interesting than any results to which their experiments
> lead.
> -- Norbert Wiener