<div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>I appreciate the help. The section view is quite extensive because each cell has 55 dofs located at the cells and on certain faces. I've appended the first of these which corresponds with the output in the first email, to save space. The following 54 are exactly the same but offset incremented by 1. (or negative 1 for negative offsets)<br></div></blockquote><div><br></div><div>Okay, from the output it is clear that this vector does not match your global section. Did you get stateVec by calling DMCreateGlobalVector()?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks for your time<br>Nicholas</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt <br><br>I apologize for any lack of clarity in the initial email.<br><br>looking at the initial output on rank 1 <br><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(235,187,255)">write</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">(</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">*</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">*</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">) </span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">"cell"</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,i,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">"offset"</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,offset,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">'oStart'</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,oStart, offset</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">-</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">oStart</span><br>cell 0 offset 2475 oStart 2640 -165<br> cell 1 offset 2530 oStart 2640 -110<br> cell 2 offset 2585 oStart 2640 -55<br> cell 3 offset 2640 oStart 2640 0<br>.....<br>cell 15 offset -771 oStart 2640 -3411<br><br><br>cell 15 provides a negative offset because it is the overlap cell (that is unowned) <br>The remained of cells are all owned. However, the first 3 cells (0,1,2) return an offset that is less than the starting ownership range. I would expect cell 0 to start at offset 2640 at minimum. <br></div></blockquote><div><br></div><div>Send the output for this section</div><div><br></div><div> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sincerely<br>Nicholas<br><br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Apologies. If it helps, there is one cell of overlap in this small test case for a 2D mesh that is 1 cell in height and a number of cells in length. .<br><br>process 0<br> Petsc VecGetLocalSize 2750<br> size(stateVecV) 2750<br><br>process 1<br> Petsc VecGetLocalSize 2640<br> size(stateVecV) 2640</div></blockquote><div><br></div><div>The offsets shown below are well-within these sizes. I do not understand the problem.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>I made a typo on the line statVecV(offset) = <set to something> in my example, I agree. (I wrote that offhand since the actual assignment is much larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's not a 1 0 indexing thing. <br><br>My question is more related to what is happening in the offsets. c0 and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed (which is why I loop from c0 to (c1-1)).<br><br>For the size inquiries. on processor 0 <br> Petsc VecGetSize(stateVec) 5390<br></div></blockquote><div><br></div><div>I need to see VecGetLocalSize()</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"> size(stateVecV) 2640<br><br>on processor 1<br>Petsc VecGetSize 5390 <br>size(stateVecV) 2750 <br><br>It's quite weird to me that processor one can have a positive offset that is less than its starting ownership index (in the initial email output). <br><br>Thanks for the assistance<br>Nicholas <br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Petsc Users, <br clear="all"><div><br>I'm working with a dmplex system with a subsampled mesh distributed with an overlap of 1. <br><br>I'm encountering unusual situations when using VecGetOwnershipRange to adjust the offset received from a global section. The logic of the following code is first to get the offset needed to index a global vector while still being able to check if it is an overlapped cell and skip if needed while counting the owned cells. <br></div></div></blockquote><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br><div style="color:rgb(255,255,255);background-color:rgb(0,36,81);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre-wrap"><div>
<div style="line-height:19px"><div><span style="color:rgb(235,187,255)">call</span> DMGetGlobalSection(dmplex,section,ierr)</div></div>
<div style="line-height:19px"><div><span style="color:rgb(235,187,255)">call</span> VecGetArrayF90(stateVec,stateVecV,ierr)</div><div><span style="color:rgb(235,187,255)">call</span> VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)</div></div><span style="color:rgb(235,187,255)">do</span> i <span style="color:rgb(153,255,255)">=</span> c0, (c1<span style="color:rgb(255,197,143)">-1</span>)</div><div> </div><div> <span style="color:rgb(235,187,255)">call</span> PetscSectionGetOffset(section,i,offset,ierr)</div><div> <span style="color:rgb(235,187,255)">write</span>(<span style="color:rgb(153,255,255)">*</span>,<span style="color:rgb(153,255,255)">*</span>) <span style="color:rgb(209,241,169)">"cell"</span>,i,<span style="color:rgb(209,241,169)">"offset"</span>,offset,<span style="color:rgb(209,241,169)">'oStart'</span>,oStart, offset<span style="color:rgb(153,255,255)">-</span>oStart</div><div> </div><div> <span style="color:rgb(235,187,255)">if</span>(offset<span style="color:rgb(153,255,255)"><</span><span style="color:rgb(255,197,143)">0</span>) <span style="color:rgb(235,187,255)">then</span></div><div> <span style="color:rgb(235,187,255)">cycle</span></div><div> <span style="color:rgb(235,187,255)">endif</span></div><div> offset<span style="color:rgb(153,255,255)">=</span>offset<span style="color:rgb(153,255,255)">-</span>oStart
</div><div style="line-height:19px"><div> plexcells<span style="color:rgb(153,255,255)">=</span>plexcells<span style="color:rgb(255,197,143)">+1</span></div></div> stateVecV(offset)= <set to something>
enddo
<div> </div></div><br>I'm noticing some very weird results that I've appended below. The GetOffset documentation notes that a negative offset indicates an unowned point (which I use to cycle). However, the offset subtraction with oStart will yield an illegal index for the Vector access. I see that on the documentation for GetOwnershipRange, it notes that this may be "ill-defined" but I wanted to see if this is type of ill-defined I can expect or there is just something terribly wrong with my PetscSection.(both the Vec and Section were produced from DMPlexDistributeField so should by definition have synchronized section information) I was wondering if there is a possible output and/or the best way to index the vector. I'm thinking of subtracting the offset of cell 0 perhaps?</div></div></blockquote><div><br></div><div>Can you show your vector sizes? Are you sure it is not the fact that F90 arrays use 1-based indices, but these are 0-based offsets?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>on rank 0<br><br> cell 0 offset 0 oStart 0 0<br> cell 1 offset 55 oStart 0 55<br> cell 2 offset 110 oStart 0 110<br> cell 3 offset 165 oStart 0 165<br> cell 4 offset 220 oStart 0 220<br> cell 5 offset 275 oStart 0 275<br> cell 6 offset 330 oStart 0 330<br> cell 7 offset 385 oStart 0 385<br> cell 8 offset 440 oStart 0 440<br> cell 9 offset 495 oStart 0 495<br> cell 10 offset 550 oStart 0 550<br> cell 11 offset 605 oStart 0 605<br> cell 12 offset 660 oStart 0 660<br> cell 13 offset 715 oStart 0 715<br><br>and on rank one <br>cell 0 offset 2475 oStart 2640 -165<br> cell 1 offset 2530 oStart 2640 -110<br> cell 2 offset 2585 oStart 2640 -55<br> cell 3 offset 2640 oStart 2640 0<br> cell 4 offset 2695 oStart 2640 55<br> cell 5 offset 2750 oStart 2640 110<br> cell 6 offset 2805 oStart 2640 165<br></div> cell 7 offset 2860 oStart 2640 220<br> cell 8 offset 2915 oStart 2640 275<br> cell 9 offset 2970 oStart 2640 330<br> cell 10 offset 3025 oStart 2640 385<br> cell 11 offset 3080 oStart 2640 440<br> cell 12 offset 3135 oStart 2640 495<br> cell 13 offset 3190 oStart 2640 550<br> cell 14 offset 3245 oStart 2640 605<br> cell 15 offset -771 oStart 2640 -3411<br><br><br>Sincerely<br>Nicholas<br><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>