<div dir="ltr"><div dir="ltr">On Mon, Dec 26, 2022 at 10:49 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Oh it's not a worry. I'm debugging this first in C++, and once it's working I don't actually need to view what's happening in Fortran when I move over. In my C debugging code. After I create the distribution vector and distribute the field based on your input, I'm adding <br><br>VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex);<br><br>But I can't compile it because VecView_Plex is undefined. Thanks<br></div></blockquote><div><br></div><div>You need</div><div><br></div><div>#include <petsc/private/dmpleximpl.h></div><div><br></div><div> Thanks</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sincerely<br>Nicholas<br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 26, 2022 at 10:45 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 26, 2022 at 10:40 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>1) I'm not sure I follow how to call this. If I insert the VecSetOperation call I'm not exactly sure what the VecView_Plex is or where it is defined? <br></div></blockquote><div><br></div><div>Shoot, this cannot be done in Fortran. I will rewrite this step to get it working for you. It should have been done anyway. I cannot do it</div><div>today since I have to grade finals, but I should have it this week.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">2) Otherwise I've solved this problem with the insight you provided into the local section. Things look good on the ASCII output but if we can resolve 1 then I think the loop is fully closed and I can just worry about the fortran translation. <br><br>Thanks again for all your help.<br><br>Sincerely<br>Nicholas<br><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 26, 2022 at 9:37 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 26, 2022 at 3:21 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Matt<br><br>I was able to get this all squared away. It turns out I was initializing the viewer incorrectly—my mistake. However, there is a follow-up question. A while back, we discussed distributing a vector field from an initial DM to a new distributed DM. The way you said to do this was<br><br><div style="color:rgb(255,255,255);background-color:rgb(0,36,81);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre-wrap"><div style="line-height:19px"><div><span style="color:rgb(114,133,183)"> // Distribute the submesh with overlap of 1 </span></div><div> <span style="color:rgb(187,218,255)">DMPlexDistribute</span>(<span style="color:rgb(255,157,164)">sub_da</span>, <span style="color:rgb(255,157,164)">overlap</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">distributionSF</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">sub_da_dist</span>);</div><div><span style="color:rgb(114,133,183)"> //Create a vector and section for the distribution </span><br></div><div> <span style="color:rgb(187,218,255)">VecCreate</span>(<span style="color:rgb(255,157,164)">PETSC_COMM_WORLD</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">state_dist</span>);</div><div> <span style="color:rgb(187,218,255)">VecSetDM</span>(<span style="color:rgb(255,157,164)">state_dist</span>, <span style="color:rgb(255,157,164)">sub_da_dist</span>);</div><div> <span style="color:rgb(187,218,255)">PetscSectionCreate</span>(<span style="color:rgb(255,157,164)">PETSC_COMM_WORLD</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">distSection</span>);</div><div> <span style="color:rgb(187,218,255)">DMSetLocalSection</span>(<span style="color:rgb(255,157,164)">sub_da_dist</span>, <span style="color:rgb(255,157,164)">distSection</span>);</div><div> <span style="color:rgb(187,218,255)">DMPlexDistributeField</span>(<span style="color:rgb(255,157,164)">sub_da_dist</span>, <span style="color:rgb(255,157,164)">distributionSF</span>, <span style="color:rgb(255,157,164)">filteredSection</span>, <span style="color:rgb(255,157,164)">state_filtered</span>, <span style="color:rgb(255,157,164)">distSection</span>, <span style="color:rgb(255,157,164)">state_dist</span>);</div></div></div> I've forgone Fortran to debug this all in C and then integrate the function calls into the Fortran code. <br><br>There are two questions here. <br><br>1) How do I associate a vector associated with a DM using VecSetDM to output properly as a VTK? When I call VecView at present, if I call VecView on state_dist, it will not output anything.<br></div></blockquote><div><br></div><div>This is a problem. The different pieces of interface were added at different times. We should really move that manipulation of the function table into VecSetDM(). Here is the code:</div><div><br></div><div> <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexcreate.c#L4135" target="_blank">https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexcreate.c#L4135</a></div><div><br></div><div>You can make the overload call yourself for now, until we decide on the best fix.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">2) The visualization is nice, but when I look at the Vec of the distributed field using stdout, something isn't distributing correctly, as the vector still has some uninitialized values. This is apparent if I output the original vector and the distributed vector. Examining the inside of DMPlexDistributeField I suspect I'm making a mistake with the sections I'm passing. filtered section in this case is the global section but if I try the local section I get an error so I'm not sure. <br></div></blockquote><div><br></div><div>These should definitely be local sections. Global sections are always built after the fact, and building the global section needs the SF that indicates what points are shared, not the distribution SF that moves points. I need to go back and put in checks that all the arguments are the right type. Thanks for bringing that up. Lets track down the error for local sections.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><b>Original Vector(state_filtered)</b><br>Vec Object: Vec_0x84000004_1 2 MPI processes<br> type: mpi<br>Process [0]<br>101325.<br>300.<br>101326.<br>301.<br>101341.<br>316.<br>Process [1]<br>101325.<br>300.<br>101326.<br>301.<br>101345.<br>320.<br>101497.<br>472.<br>101516.<br>491.<div><b>Re-Distributed Vector (state_dist) </b></div><div>Vec Object: 2 MPI processes<br> type: mpi<br>Process [0]<br>101325.<br>300.<br>101326.<br>301.<br>101341.<br>316.<br>7.90505e-323<br>1.97626e-323<br>4.30765e-312<br>6.91179e-310<br>Process [1]<br>101497.<br>472.<br>101516.<br>491.<br>1.99665e-314<br>8.14714e-321<br><br><br>Any insight on distributing this field and correcting the error would be appreciated. <br><br>Sincerely and Happy Holiday<br>Nicholas<br><br> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 23, 2022 at 10:57 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Dec 22, 2022 at 12:41 AM Nicholas Arnold-Medabalimi <<a href="mailto:narnoldm@umich.edu" target="_blank">narnoldm@umich.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Petsc Users<br><br>I've been having trouble consistently getting a vector generated from a DM to output to VTK correctly. I've used ex1.c (which works properly)to try and figure it out, but I'm still having some issues. I must be missing something small that isn't correctly associating the section with the DM. <br><br><div style="color:rgb(255,255,255);background-color:rgb(0,36,81);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre-wrap"><div> <span style="color:rgb(187,218,255)">DMPlexGetChart</span>(<span style="color:rgb(255,157,164)">dm</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">p0</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">p1</span>);</div><div> <span style="color:rgb(255,238,173)">PetscSection</span> <span style="color:rgb(255,157,164)">section_full</span>;</div><div> <span style="color:rgb(187,218,255)">PetscSectionCreate</span>(<span style="color:rgb(255,157,164)">PETSC_COMM_WORLD</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">section_full</span>);</div><div> <span style="color:rgb(187,218,255)">PetscSectionSetNumFields</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,197,143)">1</span>);</div><div> <span style="color:rgb(187,218,255)">PetscSectionSetChart</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,157,164)">p0</span>, <span style="color:rgb(255,157,164)">p1</span>);</div><div> <span style="color:rgb(187,218,255)">PetscSectionSetFieldName</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,197,143)">0</span>, <span style="color:rgb(209,241,169)">"state"</span>);</div><br><div> <span style="color:rgb(235,187,255)">for</span> (<span style="color:rgb(235,187,255)">int</span> <span style="color:rgb(255,157,164)">i</span> <span style="color:rgb(153,255,255)">=</span> <span style="color:rgb(255,157,164)">c0</span>; <span style="color:rgb(255,157,164)">i</span> <span style="color:rgb(153,255,255)"><</span> <span style="color:rgb(255,157,164)">c1</span>; <span style="color:rgb(255,157,164)">i</span><span style="color:rgb(153,255,255)">++</span>)</div><div> {</div><div> <span style="color:rgb(187,218,255)">PetscSectionSetDof</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,157,164)">i</span>, <span style="color:rgb(255,197,143)">1</span>); </div><div> <span style="color:rgb(187,218,255)">PetscSectionSetFieldDof</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,157,164)">i</span>, <span style="color:rgb(255,197,143)">0</span>, <span style="color:rgb(255,197,143)">1</span>);</div><div> }</div><div> <span style="color:rgb(187,218,255)">PetscSectionSetUp</span>(<span style="color:rgb(255,157,164)">section_full</span>);</div><div> <span style="color:rgb(187,218,255)">DMSetNumFields</span>(<span style="color:rgb(255,157,164)">dm</span>, <span style="color:rgb(255,197,143)">1</span>);</div><div> <span style="color:rgb(187,218,255)">DMSetLocalSection</span>(<span style="color:rgb(255,157,164)">dm</span>, <span style="color:rgb(255,157,164)">section_full</span>);</div><div> <span style="color:rgb(187,218,255)">DMCreateGlobalVector</span>(<span style="color:rgb(255,157,164)">dm</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">state_full</span>);</div><br><div> <span style="color:rgb(235,187,255)">int</span> <span style="color:rgb(255,157,164)">o0</span>, <span style="color:rgb(255,157,164)">o1</span>;</div><div> <span style="color:rgb(187,218,255)">VecGetOwnershipRange</span>(<span style="color:rgb(255,157,164)">state_full</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">o0</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">o1</span>);</div><div> <span style="color:rgb(255,238,173)">PetscScalar</span> <span style="color:rgb(153,255,255)">*</span><span style="color:rgb(255,157,164)">state_full_array</span>;</div><div> <span style="color:rgb(187,218,255)">VecGetArray</span>(<span style="color:rgb(255,157,164)">state_full</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">state_full_array</span>);</div><br><div> <span style="color:rgb(235,187,255)">for</span> (<span style="color:rgb(235,187,255)">int</span> <span style="color:rgb(255,157,164)">i</span> <span style="color:rgb(153,255,255)">=</span> <span style="color:rgb(255,197,143)">0</span>; <span style="color:rgb(255,157,164)">i</span> <span style="color:rgb(153,255,255)"><</span> (<span style="color:rgb(255,157,164)">c1</span> <span style="color:rgb(153,255,255)">-</span> <span style="color:rgb(255,157,164)">c0</span>); <span style="color:rgb(255,157,164)">i</span><span style="color:rgb(153,255,255)">++</span>)</div><div> {</div><div> <span style="color:rgb(235,187,255)">int</span> <span style="color:rgb(255,157,164)">offset</span>;</div><div> <span style="color:rgb(187,218,255)">PetscSectionGetOffset</span>(<span style="color:rgb(255,157,164)">section_full</span>, <span style="color:rgb(255,157,164)">i</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">offset</span>);</div><div> <span style="color:rgb(255,157,164)">state_full_array</span>[<span style="color:rgb(255,157,164)">offset</span>] <span style="color:rgb(153,255,255)">=</span> <span style="color:rgb(255,197,143)">101325</span> <span style="color:rgb(153,255,255)">+</span> <span style="color:rgb(255,157,164)">i</span>;</div><div> }</div><br><div> <span style="color:rgb(187,218,255)">VecRestoreArray</span>(<span style="color:rgb(255,157,164)">state_full</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">state_full_array</span>);</div><br><div> </div><div> <span style="color:rgb(187,218,255)">PetscViewerCreate</span>(<span style="color:rgb(255,157,164)">PETSC_COMM_WORLD</span>, <span style="color:rgb(153,255,255)">&</span><span style="color:rgb(255,157,164)">viewer</span>);</div><div> <span style="color:rgb(187,218,255)">PetscViewerSetType</span>(<span style="color:rgb(255,157,164)">viewer</span>, <span style="color:rgb(187,218,255)">PETSCVIEWERVTK</span>);</div><div> <span style="color:rgb(187,218,255)">PetscViewerFileSetMode</span>(<span style="color:rgb(255,157,164)">viewer</span>, <span style="color:rgb(255,157,164)">FILE_MODE_WRITE</span>);</div><div> <span style="color:rgb(187,218,255)">PetscViewerFileSetName</span>(<span style="color:rgb(255,157,164)">viewer</span>, <span style="color:rgb(209,241,169)">"mesh.vtu"</span>);</div><div> <span style="color:rgb(187,218,255)">VecView</span>(<span style="color:rgb(255,157,164)">state_full</span>, <span style="color:rgb(255,157,164)">viewer</span>);</div></div><br>If I run this mesh.vtu isn't generated at all. If I instead do a DMView passing the DM it will just output the mesh correctly.<br><br>Any assistance would be greatly appreciated. <br></div></blockquote><div><br></div><div>DMCreateGlobalVector() dispatches to DMCreateGlobalVector_Plex(), which resets the view method to VecView_Plex(), which should dispatch to VecView_Plex_Local_VTK(). You can verify this in the debugger, or send us code we can run to verify it.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sincerely<br>Nicholas<br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-family:arial;font-size:small"><font color="#000000">Nicholas Arnold-Medabalimi<br><br></font><span style="font-family:sans-serif;font-size:14px">Ph.D. Candidate</span><font color="#000000"><br>Computational Aeroscience Lab<br>University of Michigan</font></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>