[petsc-users] ISLocalToGlobalMapping + VecCreateGhost

[Mohammad Mirzadeh]

Hi guys,

I just did something by pure guessing which seems to work and I want
to make sure its the right thing!

I have a specific layout for my vectors that look like this

 --------------------------------------------------------------
| ghost values | local values | ghost values |
 --------------------------------------------------------------
0, 1, 2, ...        m, m+1, ...   m+n, m+n+1 ... N

which means all nodes with index [0,m) and [m+n, N) are to be treated
as ghost and all intermediate ones as local. Since PETSc stores the
ghost values at the end of ghosted vec, so far I have been manually
mapping back and forth between petsc and my application numbering.
After profiling my code, it turned out that about 15% of run-time was
just doing this mapping which is absolutely ridiculous!

Anyway, to fix this now I set up an ISLocalToGlobalMapping object
using my own application original local2global relation shown above.
Then I create the petsc vec like this:

// Create PetscVec
// We do not care about the actual global index of ghost nodes at this point
std::vector<PetscInt> ghost_nodes(N - n -1, 0);
ierr = VecCreateGhost(mpicomm, n+1, num_global, ghost_nodes.size(),
(const PetscInt*)&ghost_nodes[0], &v); CHKERRQ(ierr);


// Apply mapping
ierr = VecSetLocalToGlobalMapping(v, mapping); CHKERRQ(ierr);

After this point I do the usual VecGetArray on the vec and set the
values, but this time without any extra mapping ... my very simple
test seems to be ok. Is this the correct way of using
ISLocalToGlobalMapping?

I guess I'm suspicious because VecCreateGhost is supposed to
internally set the mapping and it is supposed to position ghost nodes
at the end of the array which I don't want it to do...

Thanks and sorry about the long email!