[petsc-users] scattering and communications

[Matthew Knepley]

On Thu, Jul 7, 2011 at 4:55 PM, Marco Cisternino <marco.cisternino at polito.it
> wrote:

> Hi,
> I would like to understand better how VecScatterBegin and VecScatterEnd
> work.
> I solve an interface elliptic problem on a Cartesian grid introducing extra
> unknowns (intersections between the interface and the grid axes).
> Therefore, my linear system is augmented with extra condition on these new
> unknowns.
> I use a DA to manage the grid, but I have to use MatCreateMPIAIJ to build
> the matrix because of the nature of the problem.
>  MatCreateMPIAIJ(proc->cart_**comm,proc->intersections[proc-**
> >rank],proc->intersections[**proc->rank],g_rows,g_cols,21,**
> PETSC_NULL,21,PETSC_NULL,&**fsolv->AA);
> VecCreateMPI(proc->cart_comm,**proc->intersections[proc->**
> rank],PETSC_DECIDE,&fsolv->**extended_P);
> VecCreateMPI(proc->cart_comm,**proc->intersections[proc->**
> rank],PETSC_DECIDE,&fsolv->**extended_RHS);
>
> where
> proc->intersections[proc->**rank] is the total number of unknowns for each
> processor  in its sub-domain (grid points + intersections).
> g_rows=g_cols is the total number of unknowns in the entire computational
> domain (grid points + intersections).
> cart_comm is a Cartesian communicator.
>
> The arrangement of the unknowns is such that every processor has the rows
> of the matrix and of extended_P(the solution) relative to the actual
> unknowns in its sub-domain.
> I solve the system and then I call VecScatterBegin and VecScatterEnd:
>
>  ierr=VecScatterCreate(fsolv->**extended_P,scatter->from,vec->**
> GP,scatter->to,&scatter->**scatt);
>  ierr=VecScatterBegin(scatter->**scatt,fsolv->extended_P,vec->**
> GP,INSERT_VALUES,SCATTER_**FORWARD);
>  ierr=VecScatterEnd(scatter->**scatt,fsolv->extended_P,vec->**
> GP,INSERT_VALUES,SCATTER_**FORWARD);
>
> in order to get in GP (made using DACreateGlobalVector(grid->da,**&vec->GP);
> ) only the solution on the grid points.
> It works, I mean I can get the right solution in GP, but the scattering
> doesn't scale at all!
> I would expect no communications during the scattering, doing what I do,
> but -log_summary shows me a number of MPI message growing with the number of
> processors.
> Every portion of GP contains only the grid points of a processor, while
> every portion of extended_P contains the same grid points plus the
> intersections in the relative sub-domain. which is the need for the
> communications doing such a scattering?
>

There is just a mismatch somewhere in your indices. It should be easy to
check the locally owned indices in
GP using


http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Vec/VecGetOwnershipRange.html

and compare to what you have.

   Matt


> I don't know if I was clear enough. Please, ask me what you need to
> understand my problem.
> Thanks a lot.
>
> Best regards,
>
>    Marco
>
> --
> Marco Cisternino
> PhD Student
> Politecnico di Torino
> Mobile:+393281189696
> Email:marco.cisternino at polito.**it <Email%3Amarco.cisternino at polito.it>
>
>


-- 
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20110707/036a5db8/attachment.htm>