On Fri, Jul 8, 2011 at 3:48 PM, Marco Cisternino <span dir="ltr"><<a href="mailto:marco.cisternino@polito.it">marco.cisternino@polito.it</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Thanks for the reply, Matt.<br>
I tried what you suggested, but I saw no inconsistency in the portion given to GP or extended_P.<br>
Moreover, I tried to see what happens giving no interface, i.e. a simple Laplace equation.<br>
In this case GP and extended_P have exactly the same structure and partitioning. And nothing changes.<br>
All the communications happens in VecScatterCreate, the actual scatter (begin and end) uses almost no time and no communications, but VecScatterCreate does.<br>
Could you explain me why?<br></blockquote><div><br></div><div>VecScatterCreate() must find out what needs to be communicated, even if that answer is nothing needs to be communicated.</div><div><br></div><div> Matt</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
The two IS I use to create the scatter context are from sequential c vectors as large as the entire computational domain. Could be this the problem?<br>
If I used smaller and local vectors to create the ISs and then the scatter context, would VecScatterCreate be more performing??<br>
Thanks a lot.<br>
<br>
Marco<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">
On Thu, Jul 7, 2011 at 4:55 PM, Marco Cisternino<<a href="mailto:marco.cisternino@polito.it" target="_blank">marco.cisternino@<u></u>polito.it</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
wrote:<br>
Hi,<br>
I would like to understand better how VecScatterBegin and VecScatterEnd<br>
work.<br>
I solve an interface elliptic problem on a Cartesian grid introducing extra<br>
unknowns (intersections between the interface and the grid axes).<br>
Therefore, my linear system is augmented with extra condition on these new<br>
unknowns.<br>
I use a DA to manage the grid, but I have to use MatCreateMPIAIJ to build<br>
the matrix because of the nature of the problem.<br>
MatCreateMPIAIJ(proc->cart_**<u></u>comm,proc->intersections[proc-<u></u>**<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
rank],proc->intersections[**<u></u>proc->rank],g_rows,g_cols,21,*<u></u>*<br>
</blockquote>
PETSC_NULL,21,PETSC_NULL,&**<u></u>fsolv->AA);<br>
VecCreateMPI(proc->cart_comm,*<u></u>*proc->intersections[proc->**<br>
rank],PETSC_DECIDE,&fsolv->**<u></u>extended_P);<br>
VecCreateMPI(proc->cart_comm,*<u></u>*proc->intersections[proc->**<br>
rank],PETSC_DECIDE,&fsolv->**<u></u>extended_RHS);<br>
<br>
where<br>
proc->intersections[proc->**<u></u>rank] is the total number of unknowns for each<br>
processor in its sub-domain (grid points + intersections).<br>
g_rows=g_cols is the total number of unknowns in the entire computational<br>
domain (grid points + intersections).<br>
cart_comm is a Cartesian communicator.<br>
<br>
The arrangement of the unknowns is such that every processor has the rows<br>
of the matrix and of extended_P(the solution) relative to the actual<br>
unknowns in its sub-domain.<br>
I solve the system and then I call VecScatterBegin and VecScatterEnd:<br>
<br>
ierr=VecScatterCreate(fsolv->*<u></u>*extended_P,scatter->from,vec-<u></u>>**<br>
GP,scatter->to,&scatter->**<u></u>scatt);<br>
ierr=VecScatterBegin(scatter-><u></u>**scatt,fsolv->extended_P,vec-<u></u>>**<br>
GP,INSERT_VALUES,SCATTER_**<u></u>FORWARD);<br>
ierr=VecScatterEnd(scatter->**<u></u>scatt,fsolv->extended_P,vec->*<u></u>*<br>
GP,INSERT_VALUES,SCATTER_**<u></u>FORWARD);<br>
<br>
in order to get in GP (made using DACreateGlobalVector(grid->da,<u></u>**&vec->GP);<br>
) only the solution on the grid points.<br>
It works, I mean I can get the right solution in GP, but the scattering<br>
doesn't scale at all!<br>
I would expect no communications during the scattering, doing what I do,<br>
but -log_summary shows me a number of MPI message growing with the number of<br>
processors.<br>
Every portion of GP contains only the grid points of a processor, while<br>
every portion of extended_P contains the same grid points plus the<br>
intersections in the relative sub-domain. which is the need for the<br>
communications doing such a scattering?<br>
<br>
</blockquote>
There is just a mismatch somewhere in your indices. It should be easy to<br>
check the locally owned indices in<br>
GP using<br>
<br>
<br>
<a href="http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/manualpages/Vec/VecGetOwnershipRange.html" target="_blank">http://www.mcs.anl.gov/petsc/<u></u>petsc-as/snapshots/petsc-dev/<u></u>docs/manualpages/Vec/<u></u>VecGetOwnershipRange.html</a><br>
<br>
and compare to what you have.<br>
<br>
Matt<br>
<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">
I don't know if I was clear enough. Please, ask me what you need to<br>
understand my problem.<br>
Thanks a lot.<br>
<br>
Best regards,<br>
<br>
Marco<br>
<br>
--<br>
Marco Cisternino<br>
PhD Student<br>
Politecnico di Torino<br>
Mobile:<a href="tel:%2B393281189696" value="+393281189696" target="_blank">+393281189696</a><br></div></div>
Email:marco.cisternino@polito.<u></u>**it<<a href="mailto:Email%253Amarco.cisternino@polito.it" target="_blank">Email%3Amarco.cisternino@<u></u>polito.it</a>><br>
<br>
<br>
</blockquote>
<br>
</blockquote><div><div></div><div class="h5">
<br>
-- <br>
Marco Cisternino<br>
PhD Student<br>
Politecnico di Torino<br>
Mobile:<a href="tel:%2B393281189696" value="+393281189696" target="_blank">+393281189696</a><br>
<a href="mailto:Email%3Amarco.cisternino@polito.it" target="_blank">Email:marco.cisternino@polito.<u></u>it</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>