<div dir="ltr">I added a workaround to avoid collective calls in <span style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">VecDuplicate. With 216 cores, it improved the code's performance by 5%. I can imagine more improvement with more cores. But more importantly, the "Reduction count" of MatSOR in -log_view is zero now instead of some misleading number.</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">BTW, it does not solve the imbalance problem of MatSOR which I am debugging.<br></span><div><span style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote">On Tue, Jun 19, 2018 at 10:58 PM, Smith, Barry F. <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Ahh yes. Since the layout can get large we wanted to avoid each vector having its own copy. <br>
<br>
But it seems sloppy since DMCreateGlobalVector_DA() does keep a new copy for each vector.<br>
<br>
We could revisit the entire VecDuplicate/<wbr>DMCreateGlobalVector() infrastructure for all the vector types including the DM ones to <br>
<br>
1) prevent collective calls <br>
2) reference the layout instead of making new ones. <br>
<br>
<br>
Barry<br>
<br>
It looks like VecDuplicate_MPI() does the right thing.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
> On Jun 19, 2018, at 10:35 PM, Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br>
> <br>
> "Smith, Barry F." <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> writes:<br>
> <br>
>> Jed,<br>
>> <br>
>> You added these two lines. Any idea why they are needed?<br>
> <br>
> That was just replacing the old interface<br>
> <br>
> <a href="https://bitbucket.org/petsc/petsc/commits/077aedaf8cb5b060ad5dab7a12035daf9d64dd75#Lsrc/dm/impls/da/dadist.cF19" rel="noreferrer" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/commits/<wbr>077aedaf8cb5b060ad5dab7a12035d<wbr>af9d64dd75#Lsrc/dm/impls/da/<wbr>dadist.cF19</a><br>
> <br>
> that you added here<br>
> <br>
> <a href="https://bitbucket.org/petsc/petsc/commits/5ead1ec26b293130cbc6f8fa84f0a9ffce38a0bf" rel="noreferrer" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/commits/<wbr>5ead1ec26b293130cbc6f8fa84f0a9<wbr>ffce38a0bf</a><br>
> <br>
>> Thanks<br>
>> <br>
>> Barry<br>
>> <br>
>> 3e08d2bebf99 src/dm/impls/da/dadist.c (Barry Smith 2012-09-18 14) PetscFunctionBegin;<br>
>> c688c0463f6c src/dm/impls/da/dadist.c (Matthew Knepley 2012-11-11 15) ierr = VecGetDM(g, &da);CHKERRQ(ierr);<br>
>> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 16) ierr = DMCreateGlobalVector(da,gg);<wbr>CHKERRQ(ierr);<br>
>> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 17) ierr = VecGetLayout(g,&map);CHKERRQ(<wbr>ierr);<br>
>> 077aedaf8cb5 src/dm/impls/da/dadist.c (Jed Brown 2013-02-11 18) ierr = VecSetLayout(*gg,map);CHKERRQ(<wbr>ierr);<br>
>> 2dcb2ebcaafe src/dm/impls/da/dadist.c (Barry Smith 2010-10-31 19) PetscFunctionReturn(0)<br>
>> <br>
>> <br>
>> If we were a bit more clever we could probably avoid all communication in VecDuplicate() but is it worth bothering unless we know it is problematic<br>
>> <br>
>>> On Jun 19, 2018, at 4:30 PM, Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov">jczhang@mcs.anl.gov</a>> wrote:<br>
>>> <br>
>>> I met an expensive VecDuplicate. See the attached call stack. VecDuplicate_MPI_DA indirectly calls MPI_Allreduce/Allgather to build the vector's layout, but it then immediately destroys it with VecGetLayout & VecSetLayout. Is it wrong?<br>
>>> src/dm/impls/da/dadist.c<br>
>>> PetscErrorCode VecDuplicate_MPI_DA(Vec g,Vec *gg)<br>
>>> {<br>
>>> PetscErrorCode ierr;<br>
>>> DM da;<br>
>>> PetscLayout map;<br>
>>> <br>
>>> PetscFunctionBegin;<br>
>>> ierr = VecGetDM(g, &da);CHKERRQ(ierr);<br>
>>> ierr = DMCreateGlobalVector(da,gg);<wbr>CHKERRQ(ierr);<br>
>>> ierr = VecGetLayout(g,&map);CHKERRQ(<wbr>ierr);<br>
>>> ierr = VecSetLayout(*gg,map);CHKERRQ(<wbr>ierr);<br>
>>> PetscFunctionReturn(0);<br>
>>> }<br>
>>> <br>
>>> --Junchao Zhang<br>
>>> <VecDup.png><br>
<br>
</div></div></blockquote></div><br></div>