<div dir="ltr">One of 6 MatSOR's generates a <span style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">VecDuplicate_MPI_DA. They are more <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">VecDuplicates in MatSOR, but some of them go to <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">VecDuplicate_MPI for reason I don't know.</span></span></span></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote">On Wed, Jun 20, 2018 at 12:52 PM, Smith, Barry F. <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
> On Jun 20, 2018, at 11:35 AM, Zhang, Junchao <<a href="mailto:jczhang@mcs.anl.gov">jczhang@mcs.anl.gov</a>> wrote:<br>
> <br>
> I added a workaround to avoid collective calls in VecDuplicate. With 216 cores, it improved the code's performance by 5%.<br>
<br>
</span> Wow<br>
<br>
Is every call to MatSOR generating a VecDuplicate/VecDestroy pair?<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
> I can imagine more improvement with more cores. But more importantly, the "Reduction count" of MatSOR in -log_view is zero now instead of some misleading number.<br>
> <br>
> BTW, it does not solve the imbalance problem of MatSOR which I am debugging.<br>
> <br>
> <br>
> --Junchao Zhang<br>
> <br>
> On Tue, Jun 19, 2018 at 10:58 PM, Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
> <br>
> Ahh yes. Since the layout can get large we wanted to avoid each vector having its own copy. <br>
> <br>
> But it seems sloppy since DMCreateGlobalVector_DA() does keep a new copy for each vector.<br>
> <br>
> We could revisit the entire VecDuplicate/<wbr>DMCreateGlobalVector() infrastructure for all the vector types including the DM ones to <br>
> <br>
> 1) prevent collective calls <br>
> 2) reference the layout instead of making new ones. <br>
> <br>
> <br>
> Barry<br>
> <br>
> It looks like VecDuplicate_MPI() does the right thing.<br>
> <br>
> <br>
> > On Jun 19, 2018, at 10:35 PM, Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br>
> > <br>
> > "Smith, Barry F." <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> writes:<br>
> > <br>
> >> Jed,<br>
> >> <br>
> >> You added these two lines. Any idea why they are needed?<br>
> > <br>
> > That was just replacing the old interface<br>
> > <br>
> > <a href="https://bitbucket.org/petsc/petsc/commits/077aedaf8cb5b060ad5dab7a12035daf9d64dd75#Lsrc/dm/impls/da/dadist.cF19" rel="noreferrer" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/commits/<wbr>077aedaf8cb5b060ad5dab7a12035d<wbr>af9d64dd75#Lsrc/dm/impls/da/<wbr>dadist.cF19</a><br>
> > <br>
> > that you added here<br>
> > <br>
> > <a href="https://bitbucket.org/petsc/petsc/commits/5ead1ec26b293130cbc6f8fa84f0a9ffce38a0bf" rel="noreferrer" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/commits/<wbr>5ead1ec26b293130cbc6f8fa84f0a9<wbr>ffce38a0bf</a><br>
> > <br>
> >> Thanks<br>
> >> <br>
> >> Barry<br>
> >> <br>
> >> 3e08d2bebf99 src/dm/impls/da/dadist.c (Barry Smith 2012-09-18 14) PetscFunctionBegin;<br>
> >> c688c0463f6c src/dm/impls/da/dadist.c (Matthew Knepley 2012-11-11 15) ierr = VecGetDM(g, &da);CHKERRQ(ierr);<br>
> >> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 16) ierr = DMCreateGlobalVector(da,gg);<wbr>CHKERRQ(ierr);<br>
> >> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 17) ierr = VecGetLayout(g,&map);CHKERRQ(<wbr>ierr);<br>
> >> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 18) ierr = VecSetLayout(*gg,map);CHKERRQ(<wbr>ierr);<br>
> >> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 19) PetscFunctionReturn(0)<br>
> >> <br>
> >> <br>
> >> If we were a bit more clever we could probably avoid all communication in VecDuplicate() but is it worth bothering unless we know it is problematic<br>
> >> <br>
> >>> On Jun 19, 2018, at 4:30 PM, Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov">jczhang@mcs.anl.gov</a>> wrote:<br>
> >>> <br>
> >>> I met an expensive VecDuplicate. See the attached call stack. VecDuplicate_MPI_DA indirectly calls MPI_Allreduce/Allgather to build the vector's layout, but it then immediately destroys it with VecGetLayout & VecSetLayout. Is it wrong?<br>
> >>> src/dm/impls/da/dadist.c<br>
> >>> PetscErrorCode VecDuplicate_MPI_DA(Vec g,Vec *gg)<br>
> >>> {<br>
> >>> PetscErrorCode ierr;<br>
> >>> DM da;<br>
> >>> PetscLayout map;<br>
> >>> <br>
> >>> PetscFunctionBegin;<br>
> >>> ierr = VecGetDM(g, &da);CHKERRQ(ierr);<br>
> >>> ierr = DMCreateGlobalVector(da,gg);<wbr>CHKERRQ(ierr);<br>
> >>> ierr = VecGetLayout(g,&map);CHKERRQ(<wbr>ierr);<br>
> >>> ierr = VecSetLayout(*gg,map);CHKERRQ(<wbr>ierr);<br>
> >>> PetscFunctionReturn(0);<br>
> >>> }<br>
> >>> <br>
> >>> --Junchao Zhang<br>
> >>> <VecDup.png><br>
> <br>
> <br>
<br>
</div></div></blockquote></div><br></div>