[petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 6, 2023 at 8:10 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>   Depending on the serial library you may not need to split the vector
> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
> global to natural and scatter to zero on the full vector, now the full
> vector is on the first rank and you can access what you need in that one
> vector if possible.
>

Does DMStag have a GlobalToNatural? Also, the serial code would have to
have identical interleaving.

  Thanks,

     Matt

> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
> coltonbryant2021 at u.northwestern.edu> wrote:
>
> Ah excellent! I was not aware of the ability to preallocate the objects
> and migrate them each time.
>
> Thanks!
> -Colton
>
> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
>> coltonbryant2021 at u.northwestern.edu> wrote:
>>
>>> Hello,
>>>
>>> I am working on a code in which a DMSTAG object is used to solve a fluid
>>> flow problem and I need to gather this flow data on a single process to
>>> interact with an existing (serial) library at each timestep of my
>>> simulation. After looking around the solution I've tried is:
>>>
>>> -use DMStagVecSplitToDMDA to extract vectors of each component of the
>>> flow
>>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>>> components naturally ordered
>>> -use VecScatterCreateToZero to set up and then do the scatter to gather
>>> on the single process
>>>
>>> Unless I'm misunderstanding something this method results in a lot of
>>> memory allocation/freeing happening at each step of the evolution and I was
>>> wondering if there is a way to directly perform such a scatter from the
>>> DMSTAG object without splitting as I'm doing here.
>>>
>>
>> 1) You can see here:
>>
>>
>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>>
>> that this function is small. You can do the DMDA creation manually, and
>> then just call DMStagMigrateVecDMDA() each time, which will not create
>> anything.
>>
>> 2) You can create the natural vector upfront, and just scatter each time.
>>
>> 3) You can create the serial vector upfront, and just scatter each time.
>>
>> This is some data movement. You can compress the g2n and 2zero scatters
>> using
>>
>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>>
>> as an optimization.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Any advice would be much appreciated!
>>>
>>> Best,
>>> Colton Bryant
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231206/8b375dc8/attachment.html>