[petsc-users] Vec Ownership ranges with Global Section Offsets
Nicholas Arnold-Medabalimi
narnoldm at umich.edu
Fri Jan 6 08:56:22 CST 2023
Apologies. If it helps, there is one cell of overlap in this small test
case for a 2D mesh that is 1 cell in height and a number of cells in
length. .
process 0
Petsc VecGetLocalSize 2750
size(stateVecV) 2750
process 1
Petsc VecGetLocalSize 2640
size(stateVecV) 2640
On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi Matt
>>
>> I made a typo on the line statVecV(offset) = <set to something> in my
>> example, I agree. (I wrote that offhand since the actual assignment is much
>> larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's
>> not a 1 0 indexing thing.
>>
>> My question is more related to what is happening in the offsets. c0 and
>> c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>> (which is why I loop from c0 to (c1-1)).
>>
>> For the size inquiries. on processor 0
>> Petsc VecGetSize(stateVec) 5390
>>
>
> I need to see VecGetLocalSize()
>
> Matt
>
>
>> size(stateVecV) 2640
>>
>> on processor 1
>> Petsc VecGetSize 5390
>> size(stateVecV) 2750
>>
>> It's quite weird to me that processor one can have a positive offset that
>> is less than its starting ownership index (in the initial email output).
>>
>> Thanks for the assistance
>> Nicholas
>>
>>
>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>> narnoldm at umich.edu> wrote:
>>>
>>>> Hi Petsc Users,
>>>>
>>>> I'm working with a dmplex system with a subsampled mesh distributed
>>>> with an overlap of 1.
>>>>
>>>> I'm encountering unusual situations when using VecGetOwnershipRange to
>>>> adjust the offset received from a global section. The logic of the
>>>> following code is first to get the offset needed to index a global vector
>>>> while still being able to check if it is an overlapped cell and skip if
>>>> needed while counting the owned cells.
>>>>
>>>
>>>
>>>>
>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>> do i = c0, (c1-1)
>>>>
>>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>>
>>>> if(offset<0) then
>>>> cycle
>>>> endif
>>>> offset=offset-oStart
>>>> plexcells=plexcells+1
>>>> stateVecV(offset)= <set to something> enddo
>>>>
>>>> I'm noticing some very weird results that I've appended below. The
>>>> GetOffset documentation notes that a negative offset indicates an unowned
>>>> point (which I use to cycle). However, the offset subtraction with oStart
>>>> will yield an illegal index for the Vector access. I see that on the
>>>> documentation for GetOwnershipRange, it notes that this may be
>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can
>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>> definition have synchronized section information) I was wondering if there
>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>> of subtracting the offset of cell 0 perhaps?
>>>>
>>>
>>> Can you show your vector sizes? Are you sure it is not the fact that F90
>>> arrays use 1-based indices, but these are 0-based offsets?
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> on rank 0
>>>>
>>>> cell 0 offset 0 oStart 0 0
>>>> cell 1 offset 55 oStart 0 55
>>>> cell 2 offset 110 oStart 0 110
>>>> cell 3 offset 165 oStart 0 165
>>>> cell 4 offset 220 oStart 0 220
>>>> cell 5 offset 275 oStart 0 275
>>>> cell 6 offset 330 oStart 0 330
>>>> cell 7 offset 385 oStart 0 385
>>>> cell 8 offset 440 oStart 0 440
>>>> cell 9 offset 495 oStart 0 495
>>>> cell 10 offset 550 oStart 0 550
>>>> cell 11 offset 605 oStart 0 605
>>>> cell 12 offset 660 oStart 0 660
>>>> cell 13 offset 715 oStart 0 715
>>>>
>>>> and on rank one
>>>> cell 0 offset 2475 oStart 2640 -165
>>>> cell 1 offset 2530 oStart 2640 -110
>>>> cell 2 offset 2585 oStart 2640 -55
>>>> cell 3 offset 2640 oStart 2640 0
>>>> cell 4 offset 2695 oStart 2640 55
>>>> cell 5 offset 2750 oStart 2640 110
>>>> cell 6 offset 2805 oStart 2640 165
>>>> cell 7 offset 2860 oStart 2640 220
>>>> cell 8 offset 2915 oStart 2640 275
>>>> cell 9 offset 2970 oStart 2640 330
>>>> cell 10 offset 3025 oStart 2640 385
>>>> cell 11 offset 3080 oStart 2640 440
>>>> cell 12 offset 3135 oStart 2640 495
>>>> cell 13 offset 3190 oStart 2640 550
>>>> cell 14 offset 3245 oStart 2640 605
>>>> cell 15 offset -771 oStart 2640 -3411
>>>>
>>>>
>>>> Sincerely
>>>> Nicholas
>>>>
>>>> --
>>>> Nicholas Arnold-Medabalimi
>>>>
>>>> Ph.D. Candidate
>>>> Computational Aeroscience Lab
>>>> University of Michigan
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
--
Nicholas Arnold-Medabalimi
Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/1f028a9c/attachment.html>
More information about the petsc-users
mailing list