[petsc-users] Vec Ownership ranges with Global Section Offsets

[Nicholas Arnold-Medabalimi]

Hi Petsc Users,

I'm working with a dmplex system with a subsampled mesh distributed with an
overlap of 1.

I'm encountering unusual situations when using VecGetOwnershipRange to
adjust the offset received from a global section. The logic of the
following code is first to get the offset needed to index a global vector
while still being able to check if it is an overlapped cell and skip if
needed while counting the owned cells.

call DMGetGlobalSection(dmplex,section,ierr)
call VecGetArrayF90(stateVec,stateVecV,ierr)
call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
do i = c0, (c1-1)

    call PetscSectionGetOffset(section,i,offset,ierr)
    write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart

    if(offset<0) then
        cycle
    endif
offset=offset-oStart
plexcells=plexcells+1
stateVecV(offset)= <set to something> enddo

I'm noticing some very weird results that I've appended below. The
GetOffset documentation notes that a negative offset indicates an unowned
point (which I use to cycle). However, the offset subtraction with oStart
will yield an illegal index for the Vector access. I see that on the
documentation for GetOwnershipRange, it notes that this may be
"ill-defined"  but I wanted to see if this is type of ill-defined I can
expect or there is just something terribly wrong with my PetscSection.(both
the Vec and Section were produced from DMPlexDistributeField so should by
definition have synchronized section information) I was wondering if there
is a possible output and/or the best way to index the vector. I'm thinking
of subtracting the offset of cell 0 perhaps?

on rank 0

 cell           0 offset           0 oStart           0           0
 cell           1 offset          55 oStart           0          55
 cell           2 offset         110 oStart           0         110
 cell           3 offset         165 oStart           0         165
 cell           4 offset         220 oStart           0         220
 cell           5 offset         275 oStart           0         275
 cell           6 offset         330 oStart           0         330
 cell           7 offset         385 oStart           0         385
 cell           8 offset         440 oStart           0         440
 cell           9 offset         495 oStart           0         495
 cell          10 offset         550 oStart           0         550
 cell          11 offset         605 oStart           0         605
 cell          12 offset         660 oStart           0         660
 cell          13 offset         715 oStart           0         715

and on rank one
cell           0 offset        2475 oStart        2640        -165
 cell           1 offset        2530 oStart        2640        -110
 cell           2 offset        2585 oStart        2640         -55
 cell           3 offset        2640 oStart        2640           0
 cell           4 offset        2695 oStart        2640          55
 cell           5 offset        2750 oStart        2640         110
 cell           6 offset        2805 oStart        2640         165
 cell           7 offset        2860 oStart        2640         220
 cell           8 offset        2915 oStart        2640         275
 cell           9 offset        2970 oStart        2640         330
 cell          10 offset        3025 oStart        2640         385
 cell          11 offset        3080 oStart        2640         440
 cell          12 offset        3135 oStart        2640         495
 cell          13 offset        3190 oStart        2640         550
 cell          14 offset        3245 oStart        2640         605
 cell          15 offset        -771 oStart        2640       -3411


Sincerely
Nicholas

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/5ff45b1b/attachment.html>