<div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt <br><br>I apologize for any lack of clarity in the initial email.<br><br>looking at the initial output on rank 1 <br><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(235,187,255)">write</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">(</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">*</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">*</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">) </span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">"cell"</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,i,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">"offset"</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,offset,</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(209,241,169)">'oStart'</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">,oStart, offset</span><span style="font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81);color:rgb(153,255,255)">-</span><span style="color:rgb(255,255,255);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre-wrap;background-color:rgb(0,36,81)">oStart</span><br>cell 0 offset 2475 oStart 2640 -165<br> cell 1 offset 2530 oStart 2640 -110<br> cell 2 offset 2585 oStart 2640 -55<br> cell 3 offset 2640 oStart 2640 0<br>.....<br>cell 15 offset -771 oStart 2640 -3411<br><br><br>cell 15 provides a negative offset because it is the overlap cell (that is unowned) <br>The remained of cells are all owned. However, the first 3 cells (0,1,2) return an offset that is less than the starting ownership range. I would expect cell 0 to start at offset 2640 at minimum. <br></div></blockquote><div><br></div><div>Send the output for this section</div><div><br></div><div> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD);</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sincerely<br>Nicholas<br><br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Apologies. If it helps, there is one cell of overlap in this small test case for a 2D mesh that is 1 cell in height and a number of cells in length. .<br><br>process 0<br> Petsc VecGetLocalSize 2750<br> size(stateVecV) 2750<br><br>process 1<br> Petsc VecGetLocalSize 2640<br> size(stateVecV) 2640</div></blockquote><div><br></div><div>The offsets shown below are well-within these sizes. I do not understand the problem.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>I made a typo on the line statVecV(offset) = <set to something> in my example, I agree. (I wrote that offhand since the actual assignment is much larger) I should be statVecV(offset+1) = <assignment> so I'm confident it's not a 1 0 indexing thing. <br><br>My question is more related to what is happening in the offsets. c0 and c1 are pulled using DMplexgetheight stratum, so they are zero-indexed (which is why I loop from c0 to (c1-1)).<br><br>For the size inquiries. on processor 0 <br> Petsc VecGetSize(stateVec) 5390<br></div></blockquote><div><br></div><div>I need to see VecGetLocalSize()</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"> size(stateVecV) 2640<br><br>on processor 1<br>Petsc VecGetSize 5390 <br>size(stateVecV) 2750 <br><br>It's quite weird to me that processor one can have a positive offset that is less than its starting ownership index (in the initial email output). <br><br>Thanks for the assistance<br>Nicholas <br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Petsc Users, <br clear="all"><div><br>I'm working with a dmplex system with a subsampled mesh distributed with an overlap of 1. <br><br>I'm encountering unusual situations when using VecGetOwnershipRange to adjust the offset received from a global section. The logic of the following code is first to get the offset needed to index a global vector while still being able to check if it is an overlapped cell and skip if needed while counting the owned cells. <br></div></div></blockquote><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br><div style="color:rgb(255,255,255);background-color:rgb(0,36,81);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre-wrap"><div>
<div style="line-height:19px"><div><span style="color:rgb(235,187,255)">call</span> DMGetGlobalSection(dmplex,section,ierr)</div></div>
<div style="line-height:19px"><div><span style="color:rgb(235,187,255)">call</span> VecGetArrayF90(stateVec,stateVecV,ierr)</div><div><span style="color:rgb(235,187,255)">call</span> VecGetOwnershipRange(stateVec,oStart,oEnd,ierr)</div></div><span style="color:rgb(235,187,255)">do</span> i <span style="color:rgb(153,255,255)">=</span> c0, (c1<span style="color:rgb(255,197,143)">-1</span>)</div><div> </div><div> <span style="color:rgb(235,187,255)">call</span> PetscSectionGetOffset(section,i,offset,ierr)</div><div> <span style="color:rgb(235,187,255)">write</span>(<span style="color:rgb(153,255,255)">*</span>,<span style="color:rgb(153,255,255)">*</span>) <span style="color:rgb(209,241,169)">"cell"</span>,i,<span style="color:rgb(209,241,169)">"offset"</span>,offset,<span style="color:rgb(209,241,169)">'oStart'</span>,oStart, offset<span style="color:rgb(153,255,255)">-</span>oStart</div><div> </div><div> <span style="color:rgb(235,187,255)">if</span>(offset<span style="color:rgb(153,255,255)"><</span><span style="color:rgb(255,197,143)">0</span>) <span style="color:rgb(235,187,255)">then</span></div><div> <span style="color:rgb(235,187,255)">cycle</span></div><div> <span style="color:rgb(235,187,255)">endif</span></div><div> offset<span style="color:rgb(153,255,255)">=</span>offset<span style="color:rgb(153,255,255)">-</span>oStart
</div><div style="line-height:19px"><div> plexcells<span style="color:rgb(153,255,255)">=</span>plexcells<span style="color:rgb(255,197,143)">+1</span></div></div> stateVecV(offset)= <set to something>
enddo
<div> </div></div><br>I'm noticing some very weird results that I've appended below. The GetOffset documentation notes that a negative offset indicates an unowned point (which I use to cycle). However, the offset subtraction with oStart will yield an illegal index for the Vector access. I see that on the documentation for GetOwnershipRange, it notes that this may be "ill-defined" but I wanted to see if this is type of ill-defined I can expect or there is just something terribly wrong with my PetscSection.(both the Vec and Section were produced from DMPlexDistributeField so should by definition have synchronized section information) I was wondering if there is a possible output and/or the best way to index the vector. I'm thinking of subtracting the offset of cell 0 perhaps?</div></div></blockquote><div><br></div><div>Can you show your vector sizes? Are you sure it is not the fact that F90 arrays use 1-based indices, but these are 0-based offsets?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>on rank 0<br><br> cell 0 offset 0 oStart 0 0<br> cell 1 offset 55 oStart 0 55<br> cell 2 offset 110 oStart 0 110<br> cell 3 offset 165 oStart 0 165<br> cell 4 offset 220 oStart 0 220<br> cell 5 offset 275 oStart 0 275<br> cell 6 offset 330 oStart 0 330<br> cell 7 offset 385 oStart 0 385<br> cell 8 offset 440 oStart 0 440<br> cell 9 offset 495 oStart 0 495<br> cell 10 offset 550 oStart 0 550<br> cell 11 offset 605 oStart 0 605<br> cell 12 offset 660 oStart 0 660<br> cell 13 offset 715 oStart 0 715<br><br>and on rank one <br>cell 0 offset 2475 oStart 2640 -165<br> cell 1 offset 2530 oStart 2640 -110<br> cell 2 offset 2585 oStart 2640 -55<br> cell 3 offset 2640 oStart 2640 0<br> cell 4 offset 2695 oStart 2640 55<br> cell 5 offset 2750 oStart 2640 110<br> cell 6 offset 2805 oStart 2640 165<br></div> cell 7 offset 2860 oStart 2640 220<br> cell 8 offset 2915 oStart 2640 275<br> cell 9 offset 2970 oStart 2640 330<br> cell 10 offset 3025 oStart 2640 385<br> cell 11 offset 3080 oStart 2640 440<br> cell 12 offset 3135 oStart 2640 495<br> cell 13 offset 3190 oStart 2640 550<br> cell 14 offset 3245 oStart 2640 605<br> cell 15 offset -771 oStart 2640 -3411<br><br><br>Sincerely<br>Nicholas<br><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>