<div dir="ltr"><div>Ok! Thanks for the reply. I'll take a look when I get a chance.</div><div><br></div><div>-Colton <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 13, 2023 at 10:48 AM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Dec 13, 2023 at 11:22 AM Colton Bryant <<a href="mailto:coltonbryant2021@u.northwestern.edu" target="_blank">coltonbryant2021@u.northwestern.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Thanks for the help last week. The suggestions made the implementation I had much cleaner. I had one follow up question. Is there a way to sort of undo this operation? I know the vec scatter can be done backwards to distribute the arrays but I didn't see an easy way to migrate the DMDA vectors back into the DMStag object.</div></div></blockquote><div><br></div><div>It is not there. However, writing it would be straightforward. I would</div><div><br></div><div> 1) Expose DMStagMigrateVecDMDA(), which is not currently public</div><div><br></div><div> 2) Add a ScatterMode argument</div><div><br></div><div> 3) Put in code that calls DMStagSetValuesStencil(), rather than GetValuesStencil()</div><div><br></div><div>We would be happy to take on MR on this, and could help in the review.</div><div><br></div><div> Thanks,</div><div><br></div><div> MAtt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks for any advice.</div><div><br></div><div>-Colton <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 6, 2023 at 8:18 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br id="m_4295085454238824198m_2818499949616553903m_-3146283625335661091lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Dec 6, 2023, at 8:35 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr">On Wed, Dec 6, 2023 at 8:10 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br></div><div> <span> </span>Depending on the serial library you may not need to split the vector into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just global to natural and scatter to zero on the full vector, now the full vector is on the first rank and you can access what you need in that one vector if possible.</div></blockquote><div><br></div><div>Does DMStag have a GlobalToNatural? </div></div></div></div></blockquote><div><br></div> Good point, it does not appear to have such a thing, though it could.</div><div><br><blockquote type="cite"><div><div dir="ltr" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div class="gmail_quote"><div>Also, the serial code would have to have identical interleaving.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><blockquote type="cite"><div>On Dec 6, 2023, at 6:37 PM, Colton Bryant <<a href="mailto:coltonbryant2021@u.northwestern.edu" target="_blank">coltonbryant2021@u.northwestern.edu</a>> wrote:</div><br><div><div dir="ltr"><div>Ah excellent! I was not aware of the ability to preallocate the objects and migrate them each time.<span> </span><br></div><div><br></div><div>Thanks!</div><div>-Colton<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <<a href="mailto:coltonbryant2021@u.northwestern.edu" target="_blank">coltonbryant2021@u.northwestern.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>I am working on a code in which a DMSTAG object is used to solve a fluid flow problem and I need to gather this flow data on a single process to interact with an existing (serial) library at each timestep of my simulation. After looking around the solution I've tried is:</div><div><br></div><div>-use DMStagVecSplitToDMDA to extract vectors of each component of the flow</div><div>-use DMDACreateNaturalVector and DMDAGlobalToNatural to get the components naturally ordered</div><div>-use VecScatterCreateToZero to set up and then do the scatter to gather on the single process</div><div><br></div><div>Unless I'm misunderstanding something this method results in a lot of memory allocation/freeing happening at each step of the evolution and I was wondering if there is a way to directly perform such a scatter from the DMSTAG object without splitting as I'm doing here.</div></div></blockquote><div><br></div><div>1) You can see here:</div><div><br></div><div><a href="https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA" target="_blank">https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA</a></div><div><br></div><div>that this function is small. You can do the DMDA creation manually, and then just call <span>DMStagMigrateVecDMDA() each time, which will not create anything.</span></div><div><span><br></span></div><div><span>2) You can create the natural vector upfront, and just scatter each time.</span></div><div><span><br></span></div><div><span>3) You can create the serial vector upfront, and just scatter each time.</span></div><div><span><br></span></div><div><span>This is some data movement. You can compress the g2n and 2zero scatters using</span></div><div><span><br></span></div><div><span> </span><a href="https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/" target="_blank">https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/</a></div><div><br></div><div>as an optimization.</div><div><br></div><div> <span> </span>Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Any advice would be much appreciated!</div><div><br></div><div>Best,</div><div>Colton Bryant<span> </span><br></div></div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">--<span> </span></span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div><br></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">--<span> </span></span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>