[petsc-users] Vec Ownership ranges with Global Section Offsets

Nicholas Arnold-Medabalimi narnoldm at umich.edu
Fri Jan 6 11:17:51 CST 2023


I am doing that as well (although not for vertex dofs). And I had it
working quite well for purely cell-associated DOFs. But I realized later
that I also wanted to transmit some DOFs associated with faces so I suspect
I'm messing something up there.

Something we discussed back on 12/26 (email subject:Getting a vector from a
DM to output VTK)  was associating a vector generated this way with the DM
so it can be more easily visualized using vtk files is

VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);

If there is an easy way to get this working in Fortran I would very much
appreciate it as it was very helpful when I was debugging in C.

Thanks
Nicholas

On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi Matt
>>
>> This was generated using the DMPlexDistributeField which we discussed a
>> while back. Everything seemed to be working fine when I only had cells dofs
>> but I recently added face dofs, which seems to have caused some issues.
>> Whats weird is that I'm feeding the same distribution SF and inputting a
>> vector and section that are consistent to the DMPlexDistributeField so I'd
>> expect the vectors and section output to be consistent. I'll take a closer
>> look at that.
>>
>
> The way I use DistributeField(), for example to distribute coordinates, is
> that I give it the migrationSF, the local coordinate section, and the local
> coordinate vector. It gives me back the new local coordinate section (which
> I set into the distributed DM), and the local coordinate vector. It seems
> like you are doing something else. Maybe the documentation is misleading.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks
>> Nicholas
>>
>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>>> narnoldm at umich.edu> wrote:
>>>
>>>> Hi Matt
>>>>
>>>> I appreciate the help. The section view is quite extensive because each
>>>> cell has 55 dofs located at the cells and on certain faces. I've appended
>>>> the first of these which corresponds with the output in the first email, to
>>>> save space. The following 54 are exactly the same but offset incremented by
>>>> 1. (or negative 1 for negative offsets)
>>>>
>>>
>>> Okay, from the output it is clear that this vector does not match your
>>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks for your time
>>>> Nicholas
>>>>
>>>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>>>>> narnoldm at umich.edu> wrote:
>>>>>
>>>>>> Hi Matt
>>>>>>
>>>>>> I apologize for any lack of clarity in the initial email.
>>>>>>
>>>>>> looking at the initial output on rank 1
>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>>>> cell           0 offset        2475 oStart        2640        -165
>>>>>>  cell           1 offset        2530 oStart        2640        -110
>>>>>>  cell           2 offset        2585 oStart        2640         -55
>>>>>>  cell           3 offset        2640 oStart        2640           0
>>>>>> .....
>>>>>> cell          15 offset        -771 oStart        2640       -3411
>>>>>>
>>>>>>
>>>>>> cell 15 provides a negative offset because it is the overlap cell
>>>>>> (that is unowned)
>>>>>> The remained of cells are all owned. However, the first 3 cells
>>>>>> (0,1,2) return an offset that is less than the starting ownership range. I
>>>>>> would expect cell 0 to start at offset 2640 at minimum.
>>>>>>
>>>>>
>>>>> Send the output for this section
>>>>>
>>>>>   call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Sincerely
>>>>>> Nicholas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>
>>>>>>>> Apologies. If it helps, there is one cell of overlap in this small
>>>>>>>> test case for a 2D mesh that is 1 cell in height and a number of cells in
>>>>>>>> length. .
>>>>>>>>
>>>>>>>> process 0
>>>>>>>>  Petsc VecGetLocalSize        2750
>>>>>>>>  size(stateVecV)        2750
>>>>>>>>
>>>>>>>> process 1
>>>>>>>>  Petsc VecGetLocalSize        2640
>>>>>>>>  size(stateVecV)        2640
>>>>>>>>
>>>>>>>
>>>>>>> The offsets shown below are well-within these sizes. I do not
>>>>>>> understand the problem.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Matt
>>>>>>>>>>
>>>>>>>>>> I made a typo on the line statVecV(offset) = <set to something>
>>>>>>>>>> in my example, I agree. (I wrote that offhand since the actual assignment
>>>>>>>>>> is much larger) I should be statVecV(offset+1) = <assignment> so I'm
>>>>>>>>>> confident it's not a 1 0 indexing thing.
>>>>>>>>>>
>>>>>>>>>> My question is more related to what is happening in the offsets.
>>>>>>>>>> c0 and c1 are pulled using DMplexgetheight stratum, so they are
>>>>>>>>>> zero-indexed (which is why I loop from c0 to (c1-1)).
>>>>>>>>>>
>>>>>>>>>> For the size inquiries. on processor 0
>>>>>>>>>>  Petsc VecGetSize(stateVec)        5390
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I need to see VecGetLocalSize()
>>>>>>>>>
>>>>>>>>>   Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>  size(stateVecV)           2640
>>>>>>>>>>
>>>>>>>>>> on processor 1
>>>>>>>>>> Petsc VecGetSize 5390
>>>>>>>>>> size(stateVecV)    2750
>>>>>>>>>>
>>>>>>>>>> It's quite weird to me that processor one can have a positive
>>>>>>>>>> offset that is less than its starting ownership index (in the initial email
>>>>>>>>>> output).
>>>>>>>>>>
>>>>>>>>>> Thanks for the assistance
>>>>>>>>>> Nicholas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>>>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Petsc Users,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm working with a dmplex system with a subsampled mesh
>>>>>>>>>>>> distributed with an overlap of 1.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm encountering unusual situations when using
>>>>>>>>>>>> VecGetOwnershipRange to adjust the offset received from a global section.
>>>>>>>>>>>> The logic of the following code is first to get the offset needed to index
>>>>>>>>>>>> a global vector while still being able to check if it is an overlapped cell
>>>>>>>>>>>> and skip if needed while counting the owned cells.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>>>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>>>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>>>>>>>>>> do i = c0, (c1-1)
>>>>>>>>>>>>
>>>>>>>>>>>>     call PetscSectionGetOffset(section,i,offset,ierr)
>>>>>>>>>>>>     write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset
>>>>>>>>>>>> -oStart
>>>>>>>>>>>>
>>>>>>>>>>>>     if(offset<0) then
>>>>>>>>>>>>         cycle
>>>>>>>>>>>>     endif
>>>>>>>>>>>> offset=offset-oStart
>>>>>>>>>>>> plexcells=plexcells+1
>>>>>>>>>>>> stateVecV(offset)= <set to something> enddo
>>>>>>>>>>>>
>>>>>>>>>>>> I'm noticing some very weird results that I've appended below.
>>>>>>>>>>>> The GetOffset documentation notes that a negative offset indicates an
>>>>>>>>>>>> unowned point (which I use to cycle). However, the offset subtraction with
>>>>>>>>>>>> oStart will yield an illegal index for the Vector access. I see that on the
>>>>>>>>>>>> documentation for GetOwnershipRange, it notes that this may be
>>>>>>>>>>>> "ill-defined"  but I wanted to see if this is type of ill-defined I can
>>>>>>>>>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>>>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>>>>>>>>>> definition have synchronized section information) I was wondering if there
>>>>>>>>>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>>>>>>>>>> of subtracting the offset of cell 0 perhaps?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you show your vector sizes? Are you sure it is not the fact
>>>>>>>>>>> that F90 arrays use 1-based indices, but these are 0-based offsets?
>>>>>>>>>>>
>>>>>>>>>>>   Thanks,
>>>>>>>>>>>
>>>>>>>>>>>      Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> on rank 0
>>>>>>>>>>>>
>>>>>>>>>>>>  cell           0 offset           0 oStart           0
>>>>>>>>>>>>   0
>>>>>>>>>>>>  cell           1 offset          55 oStart           0
>>>>>>>>>>>>  55
>>>>>>>>>>>>  cell           2 offset         110 oStart           0
>>>>>>>>>>>> 110
>>>>>>>>>>>>  cell           3 offset         165 oStart           0
>>>>>>>>>>>> 165
>>>>>>>>>>>>  cell           4 offset         220 oStart           0
>>>>>>>>>>>> 220
>>>>>>>>>>>>  cell           5 offset         275 oStart           0
>>>>>>>>>>>> 275
>>>>>>>>>>>>  cell           6 offset         330 oStart           0
>>>>>>>>>>>> 330
>>>>>>>>>>>>  cell           7 offset         385 oStart           0
>>>>>>>>>>>> 385
>>>>>>>>>>>>  cell           8 offset         440 oStart           0
>>>>>>>>>>>> 440
>>>>>>>>>>>>  cell           9 offset         495 oStart           0
>>>>>>>>>>>> 495
>>>>>>>>>>>>  cell          10 offset         550 oStart           0
>>>>>>>>>>>> 550
>>>>>>>>>>>>  cell          11 offset         605 oStart           0
>>>>>>>>>>>> 605
>>>>>>>>>>>>  cell          12 offset         660 oStart           0
>>>>>>>>>>>> 660
>>>>>>>>>>>>  cell          13 offset         715 oStart           0
>>>>>>>>>>>> 715
>>>>>>>>>>>>
>>>>>>>>>>>> and on rank one
>>>>>>>>>>>> cell           0 offset        2475 oStart        2640
>>>>>>>>>>>>  -165
>>>>>>>>>>>>  cell           1 offset        2530 oStart        2640
>>>>>>>>>>>>  -110
>>>>>>>>>>>>  cell           2 offset        2585 oStart        2640
>>>>>>>>>>>> -55
>>>>>>>>>>>>  cell           3 offset        2640 oStart        2640
>>>>>>>>>>>>   0
>>>>>>>>>>>>  cell           4 offset        2695 oStart        2640
>>>>>>>>>>>>  55
>>>>>>>>>>>>  cell           5 offset        2750 oStart        2640
>>>>>>>>>>>> 110
>>>>>>>>>>>>  cell           6 offset        2805 oStart        2640
>>>>>>>>>>>> 165
>>>>>>>>>>>>  cell           7 offset        2860 oStart        2640
>>>>>>>>>>>> 220
>>>>>>>>>>>>  cell           8 offset        2915 oStart        2640
>>>>>>>>>>>> 275
>>>>>>>>>>>>  cell           9 offset        2970 oStart        2640
>>>>>>>>>>>> 330
>>>>>>>>>>>>  cell          10 offset        3025 oStart        2640
>>>>>>>>>>>> 385
>>>>>>>>>>>>  cell          11 offset        3080 oStart        2640
>>>>>>>>>>>> 440
>>>>>>>>>>>>  cell          12 offset        3135 oStart        2640
>>>>>>>>>>>> 495
>>>>>>>>>>>>  cell          13 offset        3190 oStart        2640
>>>>>>>>>>>> 550
>>>>>>>>>>>>  cell          14 offset        3245 oStart        2640
>>>>>>>>>>>> 605
>>>>>>>>>>>>  cell          15 offset        -771 oStart        2640
>>>>>>>>>>>> -3411
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sincerely
>>>>>>>>>>>> Nicholas
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>>>>
>>>>>>>>>>>> Ph.D. Candidate
>>>>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>>>>> University of Michigan
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>>
>>>>>>>>>> Ph.D. Candidate
>>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>>> University of Michigan
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>
>>>>>>>> Ph.D. Candidate
>>>>>>>> Computational Aeroscience Lab
>>>>>>>> University of Michigan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nicholas Arnold-Medabalimi
>>>>>>
>>>>>> Ph.D. Candidate
>>>>>> Computational Aeroscience Lab
>>>>>> University of Michigan
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nicholas Arnold-Medabalimi
>>>>
>>>> Ph.D. Candidate
>>>> Computational Aeroscience Lab
>>>> University of Michigan
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/e7f9caf4/attachment-0001.html>


More information about the petsc-users mailing list