[petsc-users] DMSTAG Gathering Vector on single process

[Matthew Knepley]

On Wed, Dec 13, 2023 at 11:22 AM Colton Bryant <
coltonbryant2021 at u.northwestern.edu> wrote:

> Hi,
>
> Thanks for the help last week. The suggestions made the implementation I
> had much cleaner. I had one follow up question. Is there a way to sort of
> undo this operation? I know the vec scatter can be done backwards to
> distribute the arrays but I didn't see an easy way to migrate the DMDA
> vectors back into the DMStag object.
>

It is not there. However, writing it would be straightforward. I would

  1) Expose DMStagMigrateVecDMDA(), which is not currently public

  2) Add a ScatterMode argument

  3) Put in code that calls DMStagSetValuesStencil(), rather than
GetValuesStencil()

We would be happy to take on MR on this, and could help in the review.

  Thanks,

     MAtt


> Thanks for any advice.
>
> -Colton
>
> On Wed, Dec 6, 2023 at 8:18 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>>
>> On Dec 6, 2023, at 8:35 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Wed, Dec 6, 2023 at 8:10 PM Barry Smith <bsmith at petsc.dev> wrote:
>>
>>>
>>>   Depending on the serial library you may not need to split the vector
>>> into DMDA vectors with DMStagVecSplitToDMDA() for each component. Just
>>> global to natural and scatter to zero on the full vector, now the full
>>> vector is on the first rank and you can access what you need in that one
>>> vector if possible.
>>>
>>
>> Does DMStag have a GlobalToNatural?
>>
>>
>>    Good point, it does not appear to have such a thing, though it could.
>>
>> Also, the serial code would have to have identical interleaving.
>>
>>   Thanks,
>>
>>      Matt
>>
>>> On Dec 6, 2023, at 6:37 PM, Colton Bryant <
>>> coltonbryant2021 at u.northwestern.edu> wrote:
>>>
>>> Ah excellent! I was not aware of the ability to preallocate the objects
>>> and migrate them each time.
>>>
>>> Thanks!
>>> -Colton
>>>
>>> On Wed, Dec 6, 2023 at 5:18 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Dec 6, 2023 at 5:54 PM Colton Bryant <
>>>> coltonbryant2021 at u.northwestern.edu> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am working on a code in which a DMSTAG object is used to solve a
>>>>> fluid flow problem and I need to gather this flow data on a single process
>>>>> to interact with an existing (serial) library at each timestep of my
>>>>> simulation. After looking around the solution I've tried is:
>>>>>
>>>>> -use DMStagVecSplitToDMDA to extract vectors of each component of the
>>>>> flow
>>>>> -use DMDACreateNaturalVector and DMDAGlobalToNatural to get the
>>>>> components naturally ordered
>>>>> -use VecScatterCreateToZero to set up and then do the scatter to
>>>>> gather on the single process
>>>>>
>>>>> Unless I'm misunderstanding something this method results in a lot of
>>>>> memory allocation/freeing happening at each step of the evolution and I was
>>>>> wondering if there is a way to directly perform such a scatter from the
>>>>> DMSTAG object without splitting as I'm doing here.
>>>>>
>>>>
>>>> 1) You can see here:
>>>>
>>>>
>>>> https://petsc.org/main/src/dm/impls/stag/stagda.c.html#DMStagVecSplitToDMDA
>>>>
>>>> that this function is small. You can do the DMDA creation manually, and
>>>> then just call DMStagMigrateVecDMDA() each time, which will not create
>>>> anything.
>>>>
>>>> 2) You can create the natural vector upfront, and just scatter each
>>>> time.
>>>>
>>>> 3) You can create the serial vector upfront, and just scatter each time.
>>>>
>>>> This is some data movement. You can compress the g2n and 2zero scatters
>>>> using
>>>>
>>>>   https://petsc.org/main/manualpages/PetscSF/PetscSFCompose/
>>>>
>>>> as an optimization.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Any advice would be much appreciated!
>>>>>
>>>>> Best,
>>>>> Colton Bryant
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231213/1828b4b8/attachment.html>