[petsc-users] Vec Ownership ranges with Global Section Offsets

[Matthew Knepley]

On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi Petsc Users,
>
> I'm working with a dmplex system with a subsampled mesh distributed with
> an overlap of 1.
>
> I'm encountering unusual situations when using VecGetOwnershipRange to
> adjust the offset received from a global section. The logic of the
> following code is first to get the offset needed to index a global vector
> while still being able to check if it is an overlapped cell and skip if
> needed while counting the owned cells.
>


>
> call DMGetGlobalSection(dmplex,section,ierr)
> call VecGetArrayF90(stateVec,stateVecV,ierr)
> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
> do i = c0, (c1-1)
>
>     call PetscSectionGetOffset(section,i,offset,ierr)
>     write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>
>     if(offset<0) then
>         cycle
>     endif
> offset=offset-oStart
> plexcells=plexcells+1
> stateVecV(offset)= <set to something> enddo
>
> I'm noticing some very weird results that I've appended below. The
> GetOffset documentation notes that a negative offset indicates an unowned
> point (which I use to cycle). However, the offset subtraction with oStart
> will yield an illegal index for the Vector access. I see that on the
> documentation for GetOwnershipRange, it notes that this may be
> "ill-defined"  but I wanted to see if this is type of ill-defined I can
> expect or there is just something terribly wrong with my PetscSection.(both
> the Vec and Section were produced from DMPlexDistributeField so should by
> definition have synchronized section information) I was wondering if there
> is a possible output and/or the best way to index the vector. I'm thinking
> of subtracting the offset of cell 0 perhaps?
>

Can you show your vector sizes? Are you sure it is not the fact that F90
arrays use 1-based indices, but these are 0-based offsets?

  Thanks,

     Matt


> on rank 0
>
>  cell           0 offset           0 oStart           0           0
>  cell           1 offset          55 oStart           0          55
>  cell           2 offset         110 oStart           0         110
>  cell           3 offset         165 oStart           0         165
>  cell           4 offset         220 oStart           0         220
>  cell           5 offset         275 oStart           0         275
>  cell           6 offset         330 oStart           0         330
>  cell           7 offset         385 oStart           0         385
>  cell           8 offset         440 oStart           0         440
>  cell           9 offset         495 oStart           0         495
>  cell          10 offset         550 oStart           0         550
>  cell          11 offset         605 oStart           0         605
>  cell          12 offset         660 oStart           0         660
>  cell          13 offset         715 oStart           0         715
>
> and on rank one
> cell           0 offset        2475 oStart        2640        -165
>  cell           1 offset        2530 oStart        2640        -110
>  cell           2 offset        2585 oStart        2640         -55
>  cell           3 offset        2640 oStart        2640           0
>  cell           4 offset        2695 oStart        2640          55
>  cell           5 offset        2750 oStart        2640         110
>  cell           6 offset        2805 oStart        2640         165
>  cell           7 offset        2860 oStart        2640         220
>  cell           8 offset        2915 oStart        2640         275
>  cell           9 offset        2970 oStart        2640         330
>  cell          10 offset        3025 oStart        2640         385
>  cell          11 offset        3080 oStart        2640         440
>  cell          12 offset        3135 oStart        2640         495
>  cell          13 offset        3190 oStart        2640         550
>  cell          14 offset        3245 oStart        2640         605
>  cell          15 offset        -771 oStart        2640       -3411
>
>
> Sincerely
> Nicholas
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/60318d41/attachment.html>