about ghosted vectors and there local representation

Sun Jun 25 13:49:56 CDT 2006

Dear Barry,

Sorry for not getting your point earlier.
I made a mapping for my ghosted vector based on the code sample and things 
work as expected. 

Thanks for the help
Thomas

On Sunday 25 June 2006 17:11, Barry Smith wrote:
>    Thomas,
>
>     As I told you in my last two emails the local2global mapping you
> supply to VecSetGlobalToGlobalMapping() has to match with the ghost
> values you used when creating the ghosted vector. Otherwise you have
> two different orderings that have nothing to do with each other
> so there is NO way it could possibly work.
>
>    In the development copy of PETSc I have added the
> VecSetLocalToGlobalMapping() automatically to VecCreateGhost() so users
> need not add it. Here is a snipit of the code you can start with:
>
>    /* set local to global mapping for ghosted vector */
>    ierr = PetscMalloc((n+nghost)*sizeof(PetscInt),&indices);CHKERRQ(ierr);
>    ierr = VecGetOwnershipRange(*vv,&rstart,PETSC_NULL);CHKERRQ(ierr);
>    for (i=0; i<n; i++) {
>      indices[i] = rstart + i;
>    }
>    for (i=0; i<nghost; i++) {
>      indices[n+i] = ghosts[i];
>    }
>    ierr =
> ISLocalToGlobalMappingCreate(comm,n+nghost,indices,&ltog);CHKERRQ(ierr);
>    ierr = PetscFree(indices);CHKERRQ(ierr);
>    ierr = VecSetLocalToGlobalMapping(*vv,ltog);CHKERRQ(ierr);
>    ierr = ISLocalToGlobalMappingDestroy(ltog);CHKERRQ(ierr);
>
>   This is the only localtoglobal mapping you can set if you expect the
> ghost point stuff to work.
>
>     Barry
>
> On Sun, 25 Jun 2006, Thomas Geenen wrote:
> > oke a small example with 2 cpu's
> >
> > local2global,
> > [0] 0 0
> > [0] 1 1
> > [0] 2 2
> > [0] 3 3
> > [0] 4 4
> > [0] 5 5
> > [0] 6 6
> > [0] 7 7
> > [0] 8 8
> > [0] 9 9
> > [0] 10 10
> > [0] 11 11
> > [0] 12 12
> > [0] 13 13
> > [1] 0 0
> > [1] 1 1
> > [1] 2 14
> > [1] 3 15
> > [1] 4 16
> > [1] 5 17
> > [1] 6 18
> > [1] 7 19
> > [1] 8 20
> > [1] 9 21
> > [1] 10 22
> > [1] 11 23
> > [1] 12 24
> > [1] 13 25
> > on cpu0 nc = 14, nrows =14, nghosts=0
> > on cpu1 nc= 12, nrows=14, nghosts=2, ghosts=(0,1)
> > ierr = KSPGetSolution(ksp, &x);
> > ierr = VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD);
> > ierr = VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD);
> > ierr = VecGhostGetLocalForm(x, &lx);
> > ierr = VecView(lx, PETSC_NULL);
> > cpu0
> > -1.98507e-08
> > 1.09281e-08
> > 0.592588
> > 0.160857
> > -0.666662
> > -34.895
> > 8.93136
> > 0.499999
> > 4.22046e-08
> > 0.2963
> > -0.0497465
> > 0.666661
> > 26.8951
> > -8.93131
> > cpu1
> > -0.2963
> > 0.0497465
> > 0.666662
> > -26.8951
> > 8.93131
> > -0.499999
> > -1.70893e-08
> > -0.592588
> > -0.160857
> > -0.666661
> > 34.895
> > -8.93136
> > -1.98507e-08
> > -1.98507e-08
> >
> > 1) ghost values are at the end of local solution array lx on cpu1 I do
> > the renumering myself. would be nice if this could be done by petsc
> > automaticly 2) the first two values on cpu0 are not equal to the last two
> > at cpu1
> >
> >
> > thomas
> >
> > On Sunday 25 June 2006 15:20, you wrote:
> >> ----------  Forwarded Message  ----------
> >>
> >> Subject: Re: Fwd: Re: about ghosted vectors and there local
> >> representation Date: Thursday 22 June 2006 22:13
> >> From: Barry Smith <bsmith at mcs.anl.gov>
> >> To: petsc-users at mcs.anl.gov
> >>
> >>    Send the values of local2global, nc, ghosts, nrows and nghosts for
> >> all processors (2 I hope). Also note that nrows should equal nc +
> >> nghosts.
> >>
> >>     Barry
> >>
> >> On Thu, 22 Jun 2006, Thomas Geenen wrote:
> >>> nrows is the global number of rows
> >>> local2global is the local to global renumbering
> >>> nghosts are the number of ghosts
> >>> ghosts is the global row number i want to have local access to after
> >>> the solve
> >>>
> >>>
> >>> ISLocalToGlobalMappingCreate(PETSC_COMM_WORLD, nrows, local2global,
> >>> &ltog); VecCreateGhost(PETSC_COMM_WORLD, nc,
> >>>                            	PETSC_DECIDE, nghosts[mype], ghosts, &x);
> >>> VecSetLocalToGlobalMapping(x,ltog);
> >>> VecSetValuesLocal(x, nc, indx, x_values, INSERT_VALUES);
> >>> VecAssemblyBegin(x);
> >>> VecAssemblyEnd(x);
> >>>
> >>> solve the system
> >>>
> >>> ierr = KSPGetSolution(ksp, &x);
> >>> ierr = VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD);
> >>> ierr = VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD);
> >>> ierr = VecGhostGetLocalForm(x, &lx);
> >>>
> >>> nrows is the global number of rows
> >>> local2global is the local to global renumbering
> >>> nghosts are the number of ghosts
> >>> ghosts is the global row number i want to have local access to after
> >>> the solve
> >>>
> >>>
> >>> thomas
> >>>
> >>> On Thursday 22 June 2006 09:20, you wrote:
> >>>> ----------  Forwarded Message  ----------
> >>>>
> >>>> Subject: Re: about ghosted vectors and there local representation
> >>>> Date: Tuesday 20 June 2006 20:39
> >>>> From: Barry Smith <bsmith at mcs.anl.gov>
> >>>> To: petsc-users at mcs.anl.gov
> >>>>
> >>>>    The ltog has to exactly match the way the ghosts are provided to
> >>>> VecCreateGhost(). Are they? Please send the exact example code.
> >>>>
> >>>>     Barry
> >>>>
> >>>> On Tue, 20 Jun 2006, Thomas Geenen wrote:
> >>>>> Dear Petsc users,
> >>>>>
> >>>>> I think that I do not use the ghosted vector concept in the right
> >>>>> way.
> >>>>>
> >>>>> I create a vector with
> >>>>> ierr = VecCreateGhost(PETSC_COMM_WORLD, nc,
> >>>>>                            		 PETSC_DECIDE, nghosts, ghosts, &x);
> >>>>> The entries in ghost contain the global vector positions that I want
> >>>>> to have available locally.
> >>>>>
> >>>>> I have a local representation so i apply (after having constructed
> >>>>> the mapping of course)
> >>>>> ierr = VecSetLocalToGlobalMapping(x,ltog);
> >>>>> I now insert the values
> >>>>> ierr = VecSetValuesLocal(x, nc, indx, x_values, INSERT_VALUES);
> >>>>> VecAssemblyBegin(x);
> >>>>> VecAssemblyEnd(x);
> >>>>> I am now under the assumption that petsc divided the vector entries
> >>>>> among processes according to his global representation of the vector.
> >>>>>
> >>>>> I solve my system of equations
> >>>>> KSPSolve(ksp,b,x);
> >>>>> afterwards i do
> >>>>> ierr = KSPGetSolution(ksp, &x);
> >>>>> ierr = VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD);
> >>>>> ierr = VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD);
> >>>>> ierr = VecGhostGetLocalForm(x, &lx);
> >>>>>
> >>>>> I now expect the vector to be in the local form. That means that the
> >>>>> positions that I created as ghost points are filled with the solution
> >>>>> corresponding to the global solution. I expect that they are inserted
> >>>>> in the position in my local vector corresponding to there
> >>>>> corresponding global position.
> >>>>>
> >>>>> apparently this is not what happens.
> >>>>>
> >>>>> a small example
> >>>>> 2 cpu's
> >>>>> each cpu contains 3 vector entries however there is an overlap of 1
> >>>>> on cpu 1 the vector contains points (1, 2, 3)
> >>>>> on cpu 2 the vector contains points (3, 4, 5)
> >>>>>
> >>>>> I make position 1 on cpu 2 a ghost points with nghost=1 ghosts(3)
> >>>>> after solving and restoring I expect vector lx to contain the
> >>>>> solution as (3, 4, 5)
> >>>>>
> >>>>>
> >>>>> I hope I have explained my misunderstanding of the underlying concept
> >>>>> clear enough
> >>>>>
> >>>>> thanks for your help
> >>>>> Thomas Geenen
> >>>>
> >>>> -------------------------------------------------------
> >>
> >> -------------------------------------------------------