[petsc-users] Vec Ownership ranges with Global Section Offsets
Nicholas Arnold-Medabalimi
narnoldm at umich.edu
Fri Jan 6 09:41:07 CST 2023
Hi Matt
I appreciate the help. The section view is quite extensive because each
cell has 55 dofs located at the cells and on certain faces. I've appended
the first of these which corresponds with the output in the first email, to
save space. The following 54 are exactly the same but offset incremented by
1. (or negative 1 for negative offsets)
Thanks for your time
Nicholas
On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi Matt
>>
>> I apologize for any lack of clarity in the initial email.
>>
>> looking at the initial output on rank 1
>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>> cell 0 offset 2475 oStart 2640 -165
>> cell 1 offset 2530 oStart 2640 -110
>> cell 2 offset 2585 oStart 2640 -55
>> cell 3 offset 2640 oStart 2640 0
>> .....
>> cell 15 offset -771 oStart 2640 -3411
>>
>>
>> cell 15 provides a negative offset because it is the overlap cell (that
>> is unowned)
>> The remained of cells are all owned. However, the first 3 cells (0,1,2)
>> return an offset that is less than the starting ownership range. I would
>> expect cell 0 to start at offset 2640 at minimum.
>>
>
> Send the output for this section
>
> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>
> Thanks,
>
> Matt
>
>
>> Sincerely
>> Nicholas
>>
>>
>>
>>
>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>> narnoldm at umich.edu> wrote:
>>>
>>>> Apologies. If it helps, there is one cell of overlap in this small test
>>>> case for a 2D mesh that is 1 cell in height and a number of cells in
>>>> length. .
>>>>
>>>> process 0
>>>> Petsc VecGetLocalSize 2750
>>>> size(stateVecV) 2750
>>>>
>>>> process 1
>>>> Petsc VecGetLocalSize 2640
>>>> size(stateVecV) 2640
>>>>
>>>
>>> The offsets shown below are well-within these sizes. I do not understand
>>> the problem.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>>>> narnoldm at umich.edu> wrote:
>>>>>
>>>>>> Hi Matt
>>>>>>
>>>>>> I made a typo on the line statVecV(offset) = <set to something> in my
>>>>>> example, I agree. (I wrote that offhand since the actual assignment is much
>>>>>> larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's
>>>>>> not a 1 0 indexing thing.
>>>>>>
>>>>>> My question is more related to what is happening in the offsets. c0
>>>>>> and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed
>>>>>> (which is why I loop from c0 to (c1-1)).
>>>>>>
>>>>>> For the size inquiries. on processor 0
>>>>>> Petsc VecGetSize(stateVec) 5390
>>>>>>
>>>>>
>>>>> I need to see VecGetLocalSize()
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> size(stateVecV) 2640
>>>>>>
>>>>>> on processor 1
>>>>>> Petsc VecGetSize 5390
>>>>>> size(stateVecV) 2750
>>>>>>
>>>>>> It's quite weird to me that processor one can have a positive offset
>>>>>> that is less than its starting ownership index (in the initial email
>>>>>> output).
>>>>>>
>>>>>> Thanks for the assistance
>>>>>> Nicholas
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>
>>>>>>>> Hi Petsc Users,
>>>>>>>>
>>>>>>>> I'm working with a dmplex system with a subsampled mesh distributed
>>>>>>>> with an overlap of 1.
>>>>>>>>
>>>>>>>> I'm encountering unusual situations when using VecGetOwnershipRange
>>>>>>>> to adjust the offset received from a global section. The logic of the
>>>>>>>> following code is first to get the offset needed to index a global vector
>>>>>>>> while still being able to check if it is an overlapped cell and skip if
>>>>>>>> needed while counting the owned cells.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>>>>>> do i = c0, (c1-1)
>>>>>>>>
>>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-
>>>>>>>> oStart
>>>>>>>>
>>>>>>>> if(offset<0) then
>>>>>>>> cycle
>>>>>>>> endif
>>>>>>>> offset=offset-oStart
>>>>>>>> plexcells=plexcells+1
>>>>>>>> stateVecV(offset)= <set to something> enddo
>>>>>>>>
>>>>>>>> I'm noticing some very weird results that I've appended below. The
>>>>>>>> GetOffset documentation notes that a negative offset indicates an unowned
>>>>>>>> point (which I use to cycle). However, the offset subtraction with oStart
>>>>>>>> will yield an illegal index for the Vector access. I see that on the
>>>>>>>> documentation for GetOwnershipRange, it notes that this may be
>>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can
>>>>>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>>>>>> definition have synchronized section information) I was wondering if there
>>>>>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>>>>>> of subtracting the offset of cell 0 perhaps?
>>>>>>>>
>>>>>>>
>>>>>>> Can you show your vector sizes? Are you sure it is not the fact that
>>>>>>> F90 arrays use 1-based indices, but these are 0-based offsets?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>>> on rank 0
>>>>>>>>
>>>>>>>> cell 0 offset 0 oStart 0 0
>>>>>>>> cell 1 offset 55 oStart 0 55
>>>>>>>> cell 2 offset 110 oStart 0 110
>>>>>>>> cell 3 offset 165 oStart 0 165
>>>>>>>> cell 4 offset 220 oStart 0 220
>>>>>>>> cell 5 offset 275 oStart 0 275
>>>>>>>> cell 6 offset 330 oStart 0 330
>>>>>>>> cell 7 offset 385 oStart 0 385
>>>>>>>> cell 8 offset 440 oStart 0 440
>>>>>>>> cell 9 offset 495 oStart 0 495
>>>>>>>> cell 10 offset 550 oStart 0 550
>>>>>>>> cell 11 offset 605 oStart 0 605
>>>>>>>> cell 12 offset 660 oStart 0 660
>>>>>>>> cell 13 offset 715 oStart 0 715
>>>>>>>>
>>>>>>>> and on rank one
>>>>>>>> cell 0 offset 2475 oStart 2640 -165
>>>>>>>> cell 1 offset 2530 oStart 2640 -110
>>>>>>>> cell 2 offset 2585 oStart 2640 -55
>>>>>>>> cell 3 offset 2640 oStart 2640 0
>>>>>>>> cell 4 offset 2695 oStart 2640 55
>>>>>>>> cell 5 offset 2750 oStart 2640 110
>>>>>>>> cell 6 offset 2805 oStart 2640 165
>>>>>>>> cell 7 offset 2860 oStart 2640 220
>>>>>>>> cell 8 offset 2915 oStart 2640 275
>>>>>>>> cell 9 offset 2970 oStart 2640 330
>>>>>>>> cell 10 offset 3025 oStart 2640 385
>>>>>>>> cell 11 offset 3080 oStart 2640 440
>>>>>>>> cell 12 offset 3135 oStart 2640 495
>>>>>>>> cell 13 offset 3190 oStart 2640 550
>>>>>>>> cell 14 offset 3245 oStart 2640 605
>>>>>>>> cell 15 offset -771 oStart 2640 -3411
>>>>>>>>
>>>>>>>>
>>>>>>>> Sincerely
>>>>>>>> Nicholas
>>>>>>>>
>>>>>>>> --
>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>
>>>>>>>> Ph.D. Candidate
>>>>>>>> Computational Aeroscience Lab
>>>>>>>> University of Michigan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nicholas Arnold-Medabalimi
>>>>>>
>>>>>> Ph.D. Candidate
>>>>>> Computational Aeroscience Lab
>>>>>> University of Michigan
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nicholas Arnold-Medabalimi
>>>>
>>>> Ph.D. Candidate
>>>> Computational Aeroscience Lab
>>>> University of Michigan
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
--
Nicholas Arnold-Medabalimi
Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/c6f648c4/attachment-0001.html>
-------------- next part --------------
PetscSection Object: 2 MPI processes
type not yet set
55 fields
field 0 with 1 components
Process 0:
( 0) dim 1 offset 0
( 1) dim 1 offset 55
( 2) dim 1 offset 110
( 3) dim 1 offset 165
( 4) dim 1 offset 220
( 5) dim 1 offset 275
( 6) dim 1 offset 330
( 7) dim 1 offset 385
( 8) dim 1 offset 440
( 9) dim 1 offset 495
( 10) dim 1 offset 550
( 11) dim 1 offset 605
( 12) dim 1 offset 660
( 13) dim 1 offset 715
( 14) dim 1 offset 770
( 15) dim -2 offset -2641
( 16) dim 0 offset 825
( 17) dim 0 offset 825
( 18) dim 0 offset 825
( 19) dim 0 offset 825
( 20) dim 0 offset 825
( 21) dim 0 offset 825
( 22) dim 0 offset 825
( 23) dim 0 offset 825
( 24) dim 0 offset 825
( 25) dim 0 offset 825
( 26) dim 0 offset 825
( 27) dim 0 offset 825
( 28) dim 0 offset 825
( 29) dim 0 offset 825
( 30) dim 0 offset 825
( 31) dim 0 offset 825
( 32) dim 0 offset 825
( 33) dim 0 offset 825
( 34) dim 0 offset 825
( 35) dim 0 offset 825
( 36) dim 0 offset 825
( 37) dim 0 offset 825
( 38) dim 0 offset 825
( 39) dim 0 offset 825
( 40) dim 0 offset 825
( 41) dim 0 offset 825
( 42) dim 0 offset 825
( 43) dim 0 offset 825
( 44) dim 0 offset 825
( 45) dim 0 offset 825
( 46) dim -1 offset -3301
( 47) dim -1 offset -3301
( 48) dim -1 offset -3301
( 49) dim -1 offset -3301
( 50) dim 0 offset 825
( 51) dim 1 offset 825
( 52) dim 1 offset 880
( 53) dim 0 offset 935
( 54) dim 1 offset 935
( 55) dim 0 offset 990
( 56) dim 1 offset 990
( 57) dim 1 offset 1045
( 58) dim 1 offset 1100
( 59) dim 0 offset 1155
( 60) dim 1 offset 1155
( 61) dim 0 offset 1210
( 62) dim 1 offset 1210
( 63) dim 1 offset 1265
( 64) dim 1 offset 1320
( 65) dim 0 offset 1375
( 66) dim 1 offset 1375
( 67) dim 0 offset 1430
( 68) dim 1 offset 1430
( 69) dim 1 offset 1485
( 70) dim 1 offset 1540
( 71) dim 0 offset 1595
( 72) dim 1 offset 1595
( 73) dim 0 offset 1650
( 74) dim 1 offset 1650
( 75) dim 1 offset 1705
( 76) dim 1 offset 1760
( 77) dim 0 offset 1815
( 78) dim 1 offset 1815
( 79) dim 0 offset 1870
( 80) dim 1 offset 1870
( 81) dim 1 offset 1925
( 82) dim 1 offset 1980
( 83) dim 0 offset 2035
( 84) dim 1 offset 2035
( 85) dim 0 offset 2090
( 86) dim 1 offset 2090
( 87) dim 1 offset 2145
( 88) dim 1 offset 2200
( 89) dim 0 offset 2255
( 90) dim 1 offset 2255
( 91) dim 0 offset 2310
( 92) dim 1 offset 2310
( 93) dim 1 offset 2365
( 94) dim 1 offset 2420
( 95) dim -1 offset -3686
( 96) dim -2 offset -3686
( 97) dim -1 offset -3741
( 98) dim -2 offset -3741
Process 1:
( 0) dim 1 offset 2475
( 1) dim 1 offset 2530
( 2) dim 1 offset 2585
( 3) dim 1 offset 2640
( 4) dim 1 offset 2695
( 5) dim 1 offset 2750
( 6) dim 1 offset 2805
( 7) dim 1 offset 2860
( 8) dim 1 offset 2915
( 9) dim 1 offset 2970
( 10) dim 1 offset 3025
( 11) dim 1 offset 3080
( 12) dim 1 offset 3135
( 13) dim 1 offset 3190
( 14) dim 1 offset 3245
( 15) dim -2 offset -771
( 16) dim 0 offset 3300
( 17) dim 0 offset 3300
( 18) dim 0 offset 3300
( 19) dim 0 offset 3300
( 20) dim 0 offset 3300
( 21) dim 0 offset 3300
( 22) dim 0 offset 3300
( 23) dim 0 offset 3300
( 24) dim 0 offset 3300
( 25) dim 0 offset 3300
( 26) dim 0 offset 3300
( 27) dim 0 offset 3300
( 28) dim 0 offset 3300
( 29) dim 0 offset 3300
( 30) dim 0 offset 3300
( 31) dim 0 offset 3300
( 32) dim 0 offset 3300
( 33) dim 0 offset 3300
( 34) dim 0 offset 3300
( 35) dim 0 offset 3300
( 36) dim 0 offset 3300
( 37) dim 0 offset 3300
( 38) dim 0 offset 3300
( 39) dim 0 offset 3300
( 40) dim 0 offset 3300
( 41) dim 0 offset 3300
( 42) dim 0 offset 3300
( 43) dim 0 offset 3300
( 44) dim 0 offset 3300
( 45) dim 0 offset 3300
( 46) dim 0 offset 3300
( 47) dim 0 offset 3300
( 48) dim 0 offset 3300
( 49) dim 0 offset 3300
( 50) dim 0 offset 3300
( 51) dim 0 offset 3300
( 52) dim 0 offset 3300
( 53) dim 0 offset 3300
( 54) dim -1 offset -826
( 55) dim -1 offset -826
( 56) dim 0 offset 3300
( 57) dim 1 offset 3300
( 58) dim 1 offset 3355
( 59) dim 1 offset 3410
( 60) dim 1 offset 3465
( 61) dim 1 offset 3520
( 62) dim 0 offset 3575
( 63) dim 1 offset 3575
( 64) dim 0 offset 3630
( 65) dim 1 offset 3630
( 66) dim 0 offset 3685
( 67) dim 1 offset 3685
( 68) dim 0 offset 3740
( 69) dim 1 offset 3740
( 70) dim 1 offset 3795
( 71) dim 1 offset 3850
( 72) dim 0 offset 3905
( 73) dim 1 offset 3905
( 74) dim 0 offset 3960
( 75) dim 1 offset 3960
( 76) dim 0 offset 4015
( 77) dim 1 offset 4015
( 78) dim 0 offset 4070
( 79) dim 1 offset 4070
( 80) dim 1 offset 4125
( 81) dim 1 offset 4180
( 82) dim 0 offset 4235
( 83) dim 1 offset 4235
( 84) dim 0 offset 4290
( 85) dim 1 offset 4290
( 86) dim 1 offset 4345
( 87) dim 1 offset 4400
( 88) dim 0 offset 4455
( 89) dim 1 offset 4455
( 90) dim 0 offset 4510
( 91) dim 1 offset 4510
( 92) dim 1 offset 4565
( 93) dim 1 offset 4620
( 94) dim 0 offset 4675
( 95) dim 0 offset 4675
( 96) dim 1 offset 4675
( 97) dim 1 offset 4730
( 98) dim 0 offset 4785
( 99) dim 1 offset 4785
( 100) dim 0 offset 4840
( 101) dim 1 offset 4840
( 102) dim 1 offset 4895
( 103) dim 1 offset 4950
( 104) dim 1 offset 5005
( 105) dim -1 offset -2311
( 106) dim -2 offset -2366
( 107) dim -2 offset -2421
More information about the petsc-users
mailing list