[petsc-users] Vec Ownership ranges with Global Section Offsets
Matthew Knepley
knepley at gmail.com
Fri Jan 6 14:50:56 CST 2023
On Fri, Jan 6, 2023 at 12:18 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:
> I am doing that as well (although not for vertex dofs). And I had it
> working quite well for purely cell-associated DOFs. But I realized later
> that I also wanted to transmit some DOFs associated with faces so I suspect
> I'm messing something up there.
>
> Something we discussed back on 12/26 (email subject:Getting a vector from
> a DM to output VTK) was associating a vector generated this way with the
> DM so it can be more easily visualized using vtk files is
>
> VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);
>
> If there is an easy way to get this working in Fortran I would very much
> appreciate it as it was very helpful when I was debugging in C.
>
This is a workaround for Fortran (the problem is that Fortran cannot easily
refer to the function pointers). You can
1) Set the new section into the distributed DM
2) Make a new global or local vector
3) Call VecCopy() to move the data
The new vector will have the right viewer.
Thanks,
Matt
> Thanks
> Nicholas
>
> On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi <
>> narnoldm at umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> This was generated using the DMPlexDistributeField which we discussed a
>>> while back. Everything seemed to be working fine when I only had cells dofs
>>> but I recently added face dofs, which seems to have caused some issues.
>>> Whats weird is that I'm feeding the same distribution SF and inputting a
>>> vector and section that are consistent to the DMPlexDistributeField so I'd
>>> expect the vectors and section output to be consistent. I'll take a closer
>>> look at that.
>>>
>>
>> The way I use DistributeField(), for example to distribute coordinates,
>> is that I give it the migrationSF, the local coordinate section, and the
>> local coordinate vector. It gives me back the new local coordinate section
>> (which I set into the distributed DM), and the local coordinate vector. It
>> seems like you are doing something else. Maybe the documentation is
>> misleading.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Thanks
>>> Nicholas
>>>
>>> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <
>>>> narnoldm at umich.edu> wrote:
>>>>
>>>>> Hi Matt
>>>>>
>>>>> I appreciate the help. The section view is quite extensive because
>>>>> each cell has 55 dofs located at the cells and on certain faces. I've
>>>>> appended the first of these which corresponds with the output in the first
>>>>> email, to save space. The following 54 are exactly the same but offset
>>>>> incremented by 1. (or negative 1 for negative offsets)
>>>>>
>>>>
>>>> Okay, from the output it is clear that this vector does not match your
>>>> global section. Did you get stateVec by calling DMCreateGlobalVector()?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks for your time
>>>>> Nicholas
>>>>>
>>>>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <
>>>>>> narnoldm at umich.edu> wrote:
>>>>>>
>>>>>>> Hi Matt
>>>>>>>
>>>>>>> I apologize for any lack of clarity in the initial email.
>>>>>>>
>>>>>>> looking at the initial output on rank 1
>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart
>>>>>>> cell 0 offset 2475 oStart 2640 -165
>>>>>>> cell 1 offset 2530 oStart 2640 -110
>>>>>>> cell 2 offset 2585 oStart 2640 -55
>>>>>>> cell 3 offset 2640 oStart 2640 0
>>>>>>> .....
>>>>>>> cell 15 offset -771 oStart 2640 -3411
>>>>>>>
>>>>>>>
>>>>>>> cell 15 provides a negative offset because it is the overlap cell
>>>>>>> (that is unowned)
>>>>>>> The remained of cells are all owned. However, the first 3 cells
>>>>>>> (0,1,2) return an offset that is less than the starting ownership range. I
>>>>>>> would expect cell 0 to start at offset 2640 at minimum.
>>>>>>>
>>>>>>
>>>>>> Send the output for this section
>>>>>>
>>>>>> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> Sincerely
>>>>>>> Nicholas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <
>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>
>>>>>>>>> Apologies. If it helps, there is one cell of overlap in this small
>>>>>>>>> test case for a 2D mesh that is 1 cell in height and a number of cells in
>>>>>>>>> length. .
>>>>>>>>>
>>>>>>>>> process 0
>>>>>>>>> Petsc VecGetLocalSize 2750
>>>>>>>>> size(stateVecV) 2750
>>>>>>>>>
>>>>>>>>> process 1
>>>>>>>>> Petsc VecGetLocalSize 2640
>>>>>>>>> size(stateVecV) 2640
>>>>>>>>>
>>>>>>>>
>>>>>>>> The offsets shown below are well-within these sizes. I do not
>>>>>>>> understand the problem.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <
>>>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Matt
>>>>>>>>>>>
>>>>>>>>>>> I made a typo on the line statVecV(offset) = <set to something>
>>>>>>>>>>> in my example, I agree. (I wrote that offhand since the actual assignment
>>>>>>>>>>> is much larger) I should be statVecV(offset+1) = <assignment> so I'm
>>>>>>>>>>> confident it's not a 1 0 indexing thing.
>>>>>>>>>>>
>>>>>>>>>>> My question is more related to what is happening in the offsets.
>>>>>>>>>>> c0 and c1 are pulled using DMplexgetheight stratum, so they are
>>>>>>>>>>> zero-indexed (which is why I loop from c0 to (c1-1)).
>>>>>>>>>>>
>>>>>>>>>>> For the size inquiries. on processor 0
>>>>>>>>>>> Petsc VecGetSize(stateVec) 5390
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I need to see VecGetLocalSize()
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> size(stateVecV) 2640
>>>>>>>>>>>
>>>>>>>>>>> on processor 1
>>>>>>>>>>> Petsc VecGetSize 5390
>>>>>>>>>>> size(stateVecV) 2750
>>>>>>>>>>>
>>>>>>>>>>> It's quite weird to me that processor one can have a positive
>>>>>>>>>>> offset that is less than its starting ownership index (in the initial email
>>>>>>>>>>> output).
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the assistance
>>>>>>>>>>> Nicholas
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <
>>>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <
>>>>>>>>>>>> narnoldm at umich.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Petsc Users,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm working with a dmplex system with a subsampled mesh
>>>>>>>>>>>>> distributed with an overlap of 1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm encountering unusual situations when using
>>>>>>>>>>>>> VecGetOwnershipRange to adjust the offset received from a global section.
>>>>>>>>>>>>> The logic of the following code is first to get the offset needed to index
>>>>>>>>>>>>> a global vector while still being able to check if it is an overlapped cell
>>>>>>>>>>>>> and skip if needed while counting the owned cells.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr)
>>>>>>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr)
>>>>>>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)
>>>>>>>>>>>>> do i = c0, (c1-1)
>>>>>>>>>>>>>
>>>>>>>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr)
>>>>>>>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart,
>>>>>>>>>>>>> offset-oStart
>>>>>>>>>>>>>
>>>>>>>>>>>>> if(offset<0) then
>>>>>>>>>>>>> cycle
>>>>>>>>>>>>> endif
>>>>>>>>>>>>> offset=offset-oStart
>>>>>>>>>>>>> plexcells=plexcells+1
>>>>>>>>>>>>> stateVecV(offset)= <set to something> enddo
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm noticing some very weird results that I've appended below.
>>>>>>>>>>>>> The GetOffset documentation notes that a negative offset indicates an
>>>>>>>>>>>>> unowned point (which I use to cycle). However, the offset subtraction with
>>>>>>>>>>>>> oStart will yield an illegal index for the Vector access. I see that on the
>>>>>>>>>>>>> documentation for GetOwnershipRange, it notes that this may be
>>>>>>>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined I can
>>>>>>>>>>>>> expect or there is just something terribly wrong with my PetscSection.(both
>>>>>>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so should by
>>>>>>>>>>>>> definition have synchronized section information) I was wondering if there
>>>>>>>>>>>>> is a possible output and/or the best way to index the vector. I'm thinking
>>>>>>>>>>>>> of subtracting the offset of cell 0 perhaps?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Can you show your vector sizes? Are you sure it is not the fact
>>>>>>>>>>>> that F90 arrays use 1-based indices, but these are 0-based offsets?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Matt
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> on rank 0
>>>>>>>>>>>>>
>>>>>>>>>>>>> cell 0 offset 0 oStart 0
>>>>>>>>>>>>> 0
>>>>>>>>>>>>> cell 1 offset 55 oStart 0
>>>>>>>>>>>>> 55
>>>>>>>>>>>>> cell 2 offset 110 oStart 0
>>>>>>>>>>>>> 110
>>>>>>>>>>>>> cell 3 offset 165 oStart 0
>>>>>>>>>>>>> 165
>>>>>>>>>>>>> cell 4 offset 220 oStart 0
>>>>>>>>>>>>> 220
>>>>>>>>>>>>> cell 5 offset 275 oStart 0
>>>>>>>>>>>>> 275
>>>>>>>>>>>>> cell 6 offset 330 oStart 0
>>>>>>>>>>>>> 330
>>>>>>>>>>>>> cell 7 offset 385 oStart 0
>>>>>>>>>>>>> 385
>>>>>>>>>>>>> cell 8 offset 440 oStart 0
>>>>>>>>>>>>> 440
>>>>>>>>>>>>> cell 9 offset 495 oStart 0
>>>>>>>>>>>>> 495
>>>>>>>>>>>>> cell 10 offset 550 oStart 0
>>>>>>>>>>>>> 550
>>>>>>>>>>>>> cell 11 offset 605 oStart 0
>>>>>>>>>>>>> 605
>>>>>>>>>>>>> cell 12 offset 660 oStart 0
>>>>>>>>>>>>> 660
>>>>>>>>>>>>> cell 13 offset 715 oStart 0
>>>>>>>>>>>>> 715
>>>>>>>>>>>>>
>>>>>>>>>>>>> and on rank one
>>>>>>>>>>>>> cell 0 offset 2475 oStart 2640
>>>>>>>>>>>>> -165
>>>>>>>>>>>>> cell 1 offset 2530 oStart 2640
>>>>>>>>>>>>> -110
>>>>>>>>>>>>> cell 2 offset 2585 oStart 2640
>>>>>>>>>>>>> -55
>>>>>>>>>>>>> cell 3 offset 2640 oStart 2640
>>>>>>>>>>>>> 0
>>>>>>>>>>>>> cell 4 offset 2695 oStart 2640
>>>>>>>>>>>>> 55
>>>>>>>>>>>>> cell 5 offset 2750 oStart 2640
>>>>>>>>>>>>> 110
>>>>>>>>>>>>> cell 6 offset 2805 oStart 2640
>>>>>>>>>>>>> 165
>>>>>>>>>>>>> cell 7 offset 2860 oStart 2640
>>>>>>>>>>>>> 220
>>>>>>>>>>>>> cell 8 offset 2915 oStart 2640
>>>>>>>>>>>>> 275
>>>>>>>>>>>>> cell 9 offset 2970 oStart 2640
>>>>>>>>>>>>> 330
>>>>>>>>>>>>> cell 10 offset 3025 oStart 2640
>>>>>>>>>>>>> 385
>>>>>>>>>>>>> cell 11 offset 3080 oStart 2640
>>>>>>>>>>>>> 440
>>>>>>>>>>>>> cell 12 offset 3135 oStart 2640
>>>>>>>>>>>>> 495
>>>>>>>>>>>>> cell 13 offset 3190 oStart 2640
>>>>>>>>>>>>> 550
>>>>>>>>>>>>> cell 14 offset 3245 oStart 2640
>>>>>>>>>>>>> 605
>>>>>>>>>>>>> cell 15 offset -771 oStart 2640
>>>>>>>>>>>>> -3411
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sincerely
>>>>>>>>>>>>> Nicholas
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ph.D. Candidate
>>>>>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>>>>>> University of Michigan
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>
>>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>>>
>>>>>>>>>>> Ph.D. Candidate
>>>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>>>> University of Michigan
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>>>
>>>>>>>>> Ph.D. Candidate
>>>>>>>>> Computational Aeroscience Lab
>>>>>>>>> University of Michigan
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Nicholas Arnold-Medabalimi
>>>>>>>
>>>>>>> Ph.D. Candidate
>>>>>>> Computational Aeroscience Lab
>>>>>>> University of Michigan
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nicholas Arnold-Medabalimi
>>>>>
>>>>> Ph.D. Candidate
>>>>> Computational Aeroscience Lab
>>>>> University of Michigan
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> Nicholas Arnold-Medabalimi
>>>
>>> Ph.D. Candidate
>>> Computational Aeroscience Lab
>>> University of Michigan
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230106/d59b4366/attachment-0001.html>
More information about the petsc-users
mailing list