[petsc-users] Vec Ownership ranges with Global Section Offsets
Matthew Knepley
knepley at gmail.com
Fri Jan 6 09:23:25 CST 2023
On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:
> Hi Matt
>
> I apologize for any lack of clarity in the initial email.
>
> looking at the initial output on rank 1
> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
> cell 0 offset 2475 oStart 2640 -165
> cell 1 offset 2530 oStart 2640 -110
> cell 2 offset 2585 oStart 2640 -55
> cell 3 offset 2640 oStart 2640 0
> .....
> cell 15 offset -771 oStart 2640 -3411
>
>
> cell 15 provides a negative offset because it is the overlap cell (that is
> unowned)
> The remained of cells are all owned. However, the first 3 cells (0,1,2)
> return an offset that is less than the starting ownership range. I would
> expect cell 0 to start at offset 2640 at minimum.
>
Send the output for this section
call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
Thanks,
Matt
> Sincerely
> Nicholas
>
>
>
>
> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>> narnoldm at umich.edu> wrote:
>>
>>> Apologies. If it helps, there is one cell of overlap in this small test
>>> case for a 2D mesh that is 1 cell in height and a number of cells in
>>> length. .
>>>
>>> process 0
>>> Petsc VecGetLocalSize 2750
>>> size(stateVecV) 2750
>>>
>>> process 1
>>> Petsc VecGetLocalSize 2640
>>> size(stateVecV) 2640
>>>
>>
>> The offsets shown below are well-within these sizes. I do not understand
>> the problem.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>>> narnoldm at umich.edu> wrote:
>>>>
>>>>> Hi Matt
>>>>>
>>>>> I made a typo on the line statVecV(offset) = <set to something> in my
>>>>> example, I agree. (I wrote that offhand since the actual assignment is much
>>>>> larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's
>>>>> not a 1 0 indexing thing.
>>>>>
>>>>> My question is more related to what is happening in the offsets. c0
>>>>> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>>>>> (which is why I loop from c0 to (c1-1)).
>>>>>
>>>>> For the size inquiries. on processor 0
>>>>> Petsc VecGetSize(stateVec) 5390
>>>>>
>>>>
>>>> I need to see VecGetLocalSize()
>>>>
>>>> Matt
>>>>
>>>>
>>>>> size(stateVecV) 2640
>>>>>
>>>>> on processor 1
>>>>> Petsc VecGetSize 5390
>>>>> size(stateVecV) 2750
>>>>>
>>>>> It's quite weird to me that processor one can have a positive offset
>>>>> that is less than its starting ownership index (in the initial email
>>>>> output).
>>>>>
>>>>> Thanks for the assistance
>>>>> Nicholas
>>>>>
>>>>>
>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>>>>> narnoldm at umich.edu> wrote:
>>>>>>
>>>>>>> Hi Petsc Users,
>>>>>>>
>>>>>>> I'm working with a dmplex system with a subsampled mesh distributed
>>>>>>> with an overlap of 1.
>>>>>>>
>>>>>>> I'm encountering unusual situations when using VecGetOwnershipRange
>>>>>>> to adjust the offset received from a global section. The logic of the
>>>>>>> following code is first to get the offset needed to index a global vector
>>>>>>> while still being able to check if it is an overlapped cell and skip if
>>>>>>> needed while counting the owned cells.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>>>>> do i = c0, (c1-1)
>>>>>>>
>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>>>>>>> oStart
>>>>>>>
>>>>>>> if(offset<0) then
>>>>>>> cycle
>>>>>>> endif
>>>>>>> offset=offset-oStart
>>>>>>> plexcells=plexcells+1
>>>>>>> stateVecV(offset)= <set to something> enddo
>>>>>>>
>>>>>>> I'm noticing some very weird results that I've appended below. The
>>>>>>> GetOffset documentation notes that a negative offset indicates an unowned
>>>>>>> point (which I use to cycle). However, the offset subtraction with oStart
>>>>>>> will yield an illegal index for the Vector access. I see that on the
>>>>>>> documentation for GetOwnershipRange, it notes that this may be
>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can
>>>>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>>>>> definition have synchronized section information) I was wondering if there
>>>>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>>>>> of subtracting the offset of cell 0 perhaps?
>>>>>>>
>>>>>>
>>>>>> Can you show your vector sizes? Are you sure it is not the fact that
>>>>>> F90 arrays use 1-based indices, but these are 0-based offsets?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> on rank 0
>>>>>>>
>>>>>>> cell 0 offset 0 oStart 0 0
>>>>>>> cell 1 offset 55 oStart 0 55
>>>>>>> cell 2 offset 110 oStart 0 110
>>>>>>> cell 3 offset 165 oStart 0 165
>>>>>>> cell 4 offset 220 oStart 0 220
>>>>>>> cell 5 offset 275 oStart 0 275
>>>>>>> cell 6 offset 330 oStart 0 330
>>>>>>> cell 7 offset 385 oStart 0 385
>>>>>>> cell 8 offset 440 oStart 0 440
>>>>>>> cell 9 offset 495 oStart 0 495
>>>>>>> cell 10 offset 550 oStart 0 550
>>>>>>> cell 11 offset 605 oStart 0 605
>>>>>>> cell 12 offset 660 oStart 0 660
>>>>>>> cell 13 offset 715 oStart 0 715
>>>>>>>
>>>>>>> and on rank one
>>>>>>> cell 0 offset 2475 oStart 2640 -165
>>>>>>> cell 1 offset 2530 oStart 2640 -110
>>>>>>> cell 2 offset 2585 oStart 2640 -55
>>>>>>> cell 3 offset 2640 oStart 2640 0
>>>>>>> cell 4 offset 2695 oStart 2640 55
>>>>>>> cell 5 offset 2750 oStart 2640 110
>>>>>>> cell 6 offset 2805 oStart 2640 165
>>>>>>> cell 7 offset 2860 oStart 2640 220
>>>>>>> cell 8 offset 2915 oStart 2640 275
>>>>>>> cell 9 offset 2970 oStart 2640 330
>>>>>>> cell 10 offset 3025 oStart 2640 385
>>>>>>> cell 11 offset 3080 oStart 2640 440
>>>>>>> cell 12 offset 3135 oStart 2640 495
>>>>>>> cell 13 offset 3190 oStart 2640 550
>>>>>>> cell 14 offset 3245 oStart 2640 605
>>>>>>> cell 15 offset -771 oStart 2640 -3411
>>>>>>>
>>>>>>>
>>>>>>> Sincerely
>>>>>>> Nicholas
>>>>>>>
>>>>>>> --
>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>
>>>>>>> Ph.D. Candidate
>>>>>>> Computational Aeroscience Lab
>>>>>>> University of Michigan
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nicholas Arnold-Medabalimi
>>>>>
>>>>> Ph.D. Candidate
>>>>> Computational Aeroscience Lab
>>>>> University of Michigan
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> Nicholas Arnold-Medabalimi
>>>
>>> Ph.D. Candidate
>>> Computational Aeroscience Lab
>>> University of Michigan
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/e583cd25/attachment-0001.html>
More information about the petsc-users
mailing list