[petsc-users] Vec Ownership ranges with Global Section Offsets
Nicholas Arnold-Medabalimi
narnoldm at umich.edu
Fri Jan 6 10:31:59 CST 2023
Hi Matt
This was generated using the DMPlexDistributeField which we discussed a
while back. Everything seemed to be working fine when I only had cells dofs
but I recently added face dofs, which seems to have caused some issues.
Whats weird is that I'm feeding the same distribution SF and inputting a
vector and section that are consistent to the DMPlexDistributeField so I'd
expect the vectors and section output to be consistent. I'll take a closer
look at that.
Thanks
Nicholas
On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi Matt
>>
>> I appreciate the help. The section view is quite extensive because each
>> cell has 55 dofs located at the cells and on certain faces. I've appended
>> the first of these which corresponds with the output in the first email, to
>> save space. The following 54 are exactly the same but offset incremented by
>> 1. (or negative 1 for negative offsets)
>>
>
> Okay, from the output it is clear that this vector does not match your
> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>
> Thanks,
>
> Matt
>
>
>> Thanks for your time
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>>> narnoldm at umich.edu> wrote:
>>>
>>>> Hi Matt
>>>>
>>>> I apologize for any lack of clarity in the initial email.
>>>>
>>>> looking at the initial output on rank 1
>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>> cell 0 offset 2475 oStart 2640 -165
>>>> cell 1 offset 2530 oStart 2640 -110
>>>> cell 2 offset 2585 oStart 2640 -55
>>>> cell 3 offset 2640 oStart 2640 0
>>>> .....
>>>> cell 15 offset -771 oStart 2640 -3411
>>>>
>>>>
>>>> cell 15 provides a negative offset because it is the overlap cell (that
>>>> is unowned)
>>>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>>>> return an offset that is less than the starting ownership range. I would
>>>> expect cell 0 to start at offset 2640 at minimum.
>>>>
>>>
>>> Send the output for this section
>>>
>>> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Sincerely
>>>> Nicholas
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>>>> narnoldm at umich.edu> wrote:
>>>>>
>>>>>> Apologies. If it helps, there is one cell of overlap in this small
>>>>>> test case for a 2D mesh that is 1 cell in height and a number of cells in
>>>>>> length. .
>>>>>>
>>>>>> process 0
>>>>>> Petsc VecGetLocalSize 2750
>>>>>> size(stateVecV) 2750
>>>>>>
>>>>>> process 1
>>>>>> Petsc VecGetLocalSize 2640
>>>>>> size(stateVecV) 2640
>>>>>>
>>>>>
>>>>> The offsets shown below are well-within these sizes. I do not
>>>>> understand the problem.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>
>>>>>>>> Hi Matt
>>>>>>>>
>>>>>>>> I made a typo on the line statVecV(offset) = <set to something> in
>>>>>>>> my example, I agree. (I wrote that offhand since the actual assignment is
>>>>>>>> much larger) I should be statVecV(offset+1) = <assignment> so I'm confident
>>>>>>>> it's not a 1 0 indexing thing.
>>>>>>>>
>>>>>>>> My question is more related to what is happening in the offsets. c0
>>>>>>>> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>>>>>>>> (which is why I loop from c0 to (c1-1)).
>>>>>>>>
>>>>>>>> For the size inquiries. on processor 0
>>>>>>>> Petsc VecGetSize(stateVec) 5390
>>>>>>>>
>>>>>>>
>>>>>>> I need to see VecGetLocalSize()
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>>> size(stateVecV) 2640
>>>>>>>>
>>>>>>>> on processor 1
>>>>>>>> Petsc VecGetSize 5390
>>>>>>>> size(stateVecV) 2750
>>>>>>>>
>>>>>>>> It's quite weird to me that processor one can have a positive
>>>>>>>> offset that is less than its starting ownership index (in the initial email
>>>>>>>> output).
>>>>>>>>
>>>>>>>> Thanks for the assistance
>>>>>>>> Nicholas
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Petsc Users,
>>>>>>>>>>
>>>>>>>>>> I'm working with a dmplex system with a subsampled mesh
>>>>>>>>>> distributed with an overlap of 1.
>>>>>>>>>>
>>>>>>>>>> I'm encountering unusual situations when using
>>>>>>>>>> VecGetOwnershipRange to adjust the offset received from a global section.
>>>>>>>>>> The logic of the following code is first to get the offset needed to index
>>>>>>>>>> a global vector while still being able to check if it is an overlapped cell
>>>>>>>>>> and skip if needed while counting the owned cells.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>>>>>>>> do i = c0, (c1-1)
>>>>>>>>>>
>>>>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>>>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>>>>>>>>>> oStart
>>>>>>>>>>
>>>>>>>>>> if(offset<0) then
>>>>>>>>>> cycle
>>>>>>>>>> endif
>>>>>>>>>> offset=offset-oStart
>>>>>>>>>> plexcells=plexcells+1
>>>>>>>>>> stateVecV(offset)= <set to something> enddo
>>>>>>>>>>
>>>>>>>>>> I'm noticing some very weird results that I've appended below.
>>>>>>>>>> The GetOffset documentation notes that a negative offset indicates an
>>>>>>>>>> unowned point (which I use to cycle). However, the offset subtraction with
>>>>>>>>>> oStart will yield an illegal index for the Vector access. I see that on the
>>>>>>>>>> documentation for GetOwnershipRange, it notes that this may be
>>>>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can
>>>>>>>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>>>>>>>> definition have synchronized section information) I was wondering if there
>>>>>>>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>>>>>>>> of subtracting the offset of cell 0 perhaps?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you show your vector sizes? Are you sure it is not the fact
>>>>>>>>> that F90 arrays use 1-based indices, but these are 0-based offsets?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> on rank 0
>>>>>>>>>>
>>>>>>>>>> cell 0 offset 0 oStart 0
>>>>>>>>>> 0
>>>>>>>>>> cell 1 offset 55 oStart 0
>>>>>>>>>> 55
>>>>>>>>>> cell 2 offset 110 oStart 0
>>>>>>>>>> 110
>>>>>>>>>> cell 3 offset 165 oStart 0
>>>>>>>>>> 165
>>>>>>>>>> cell 4 offset 220 oStart 0
>>>>>>>>>> 220
>>>>>>>>>> cell 5 offset 275 oStart 0
>>>>>>>>>> 275
>>>>>>>>>> cell 6 offset 330 oStart 0
>>>>>>>>>> 330
>>>>>>>>>> cell 7 offset 385 oStart 0
>>>>>>>>>> 385
>>>>>>>>>> cell 8 offset 440 oStart 0
>>>>>>>>>> 440
>>>>>>>>>> cell 9 offset 495 oStart 0
>>>>>>>>>> 495
>>>>>>>>>> cell 10 offset 550 oStart 0
>>>>>>>>>> 550
>>>>>>>>>> cell 11 offset 605 oStart 0
>>>>>>>>>> 605
>>>>>>>>>> cell 12 offset 660 oStart 0
>>>>>>>>>> 660
>>>>>>>>>> cell 13 offset 715 oStart 0
>>>>>>>>>> 715
>>>>>>>>>>
>>>>>>>>>> and on rank one
>>>>>>>>>> cell 0 offset 2475 oStart 2640 -165
>>>>>>>>>> cell 1 offset 2530 oStart 2640
>>>>>>>>>> -110
>>>>>>>>>> cell 2 offset 2585 oStart 2640
>>>>>>>>>> -55
>>>>>>>>>> cell 3 offset 2640 oStart 2640
>>>>>>>>>> 0
>>>>>>>>>> cell 4 offset 2695 oStart 2640
>>>>>>>>>> 55
>>>>>>>>>> cell 5 offset 2750 oStart 2640
>>>>>>>>>> 110
>>>>>>>>>> cell 6 offset 2805 oStart 2640
>>>>>>>>>> 165
>>>>>>>>>> cell 7 offset 2860 oStart 2640
>>>>>>>>>> 220
>>>>>>>>>> cell 8 offset 2915 oStart 2640
>>>>>>>>>> 275
>>>>>>>>>> cell 9 offset 2970 oStart 2640
>>>>>>>>>> 330
>>>>>>>>>> cell 10 offset 3025 oStart 2640
>>>>>>>>>> 385
>>>>>>>>>> cell 11 offset 3080 oStart 2640
>>>>>>>>>> 440
>>>>>>>>>> cell 12 offset 3135 oStart 2640
>>>>>>>>>> 495
>>>>>>>>>> cell 13 offset 3190 oStart 2640
>>>>>>>>>> 550
>>>>>>>>>> cell 14 offset 3245 oStart 2640
>>>>>>>>>> 605
>>>>>>>>>> cell 15 offset -771 oStart 2640
>>>>>>>>>> -3411
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sincerely
>>>>>>>>>> Nicholas
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>>
>>>>>>>>>> Ph.D. Candidate
>>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>>> University of Michigan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>
>>>>>>>> Ph.D. Candidate
>>>>>>>> Computational Aeroscience Lab
>>>>>>>> University of Michigan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nicholas Arnold-Medabalimi
>>>>>>
>>>>>> Ph.D. Candidate
>>>>>> Computational Aeroscience Lab
>>>>>> University of Michigan
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nicholas Arnold-Medabalimi
>>>>
>>>> Ph.D. Candidate
>>>> Computational Aeroscience Lab
>>>> University of Michigan
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
--
Nicholas Arnold-Medabalimi
Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/4516a1ae/attachment-0001.html>
More information about the petsc-users
mailing list