<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Hi, </div><div style="font-family: Arial, sans-serif; font-size: 14px;">Sorry for my late response. I tried with your suggestions and I think I made a progress. But I still got issues. Let me explain my latest mesh routine:</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div><ol data-editing-info="{"orderedStyleType":3,"unorderedStyleType":1}" data-listchain="__List_Chain_409"><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "1) ";"><span><span>DMPlexCreateBoxMesh</span><br></span></li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "2) ";"><span><span>DMSetFromOptions</span></span></li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "3) ";"><span><span>PetscSectionCreate</span></span></li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "4) ";">PetscSectionSetNumFields</li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "5) ";"><span style="text-decoration: none; display: inline !important; background-color: rgb(255, 255, 255);">PetscSectionSetFieldDof</span><br></li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "6) ";"><span style="text-decoration: none; animation-delay: 1ms !important; animation-duration: 0.001ms !important; animation-iteration-count: 1 !important; transition-duration: 0.01ms !important; display: inline !important; background-color: rgb(255, 255, 255);">PetscSectionSetDof</span><br></li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "7) ";">PetscSectionSetUp</li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "8) ";">DMSetLocalSection</li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "9) ";">DMSetAdjacency</li><li style="font-family: Arial, sans-serif; font-size: 14px; list-style-type: "10) ";">DMPlexDistribute</li></ol><div style="font-family:Arial, sans-serif;font-size:14px;line-height:30px"></div><div style="font-family:Arial, sans-serif;font-size:14px;line-height:30px"><br></div></div><div style="line-height: 30px;"><font face="Arial, sans-serif">It's still not working but it's promising, if I call DMPlexGetDepthStratum for cells, I can see that after distribution processors have more cells. But I couldn't figure out how to decide where the ghost/processor boundary cells start. In older mails I saw there is a function <span style="text-decoration: none; display: inline !important; background-color: rgb(255, 255, 255);">DMPlexGetHybridBounds but I think that function is deprecated. I tried to use, </span><span style="animation-delay:1ms !important;animation-duration:0.001ms !important;animation-iteration-count:1 !important;transition-duration:0.01ms !important;text-decoration:none"><span style="animation-delay:1ms !important;animation-duration:0.001ms !important;animation-iteration-count:1 !important;transition-duration:0.01ms !important">DMPlexGetCellTypeStratum</span><span> </span>as in ts/tutorials/ex11_sa.c but I'm getting -1 as cEndInterior before and after distribution. I tried it for <span>DM_POLYTOPE_FV_GHOST, </span><span>DM_POLYTOPE_INTERIOR_GHOST polytope types. I also tried calling DMPlexComputeCellTypes before DMPlexGetCellTypeStratum but nothing changed. I think I can calculate the ghost cell indices using cStart/cEnd before & after distribution but I think there is a better way I'm currently missing.</span></span></font></div><div style="line-height: 30px;"><br></div><div style="line-height: 30px;"><font face="Arial, sans-serif"><span style="animation-delay:1ms !important;animation-duration:0.001ms !important;animation-iteration-count:1 !important;transition-duration:0.01ms !important;text-decoration:none"><span>Thanks again,</span></span></font></div><div style="line-height: 30px;"><font face="Arial, sans-serif"><span style="animation-delay:1ms !important;animation-duration:0.001ms !important;animation-iteration-count:1 !important;transition-duration:0.01ms !important;text-decoration:none"><span>Guer.</span></span></font></div>
<div style="font-family: Arial, sans-serif; font-size: 14px;" class="protonmail_signature_block">
<div class="protonmail_signature_block-user protonmail_signature_block-empty">
</div>
<div class="protonmail_signature_block-proton">
</div>
</div>
<div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div class="protonmail_quote">
------- Original Message -------<br>
On Thursday, September 28th, 2023 at 10:42 PM, Matthew Knepley <knepley@gmail.com> wrote:<br><br>
<blockquote type="cite" class="protonmail_quote">
<div dir="ltr"><div dir="ltr">On Thu, Sep 28, 2023 at 3:38 PM erdemguer via petsc-users <<a target="_blank" rel="noreferrer nofollow noopener" href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px"><div style="font-size: 16px; white-space: pre-wrap; text-decoration: none; border: 0px solid rgb(217, 217, 227); box-sizing: border-box; margin: 20px 0px; font-family: Söhne, ui-sans-serif, system-ui, -apple-system, "Segoe UI", Roboto, Ubuntu, Cantarell, "Noto Sans", sans-serif, "Helvetica Neue", Arial, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji"; color: rgb(209, 213, 219);"><span></span></div><span style="font-family:"Trebuchet MS",sans-serif">Hi,</span><div><br></div><div><span style="font-family:"Trebuchet MS",sans-serif">I am currently using DMPlex in my code. It runs serially at the moment, but I'm interested in adding parallel options. Here is my workflow:</span></div><div><br></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create a DMPlex mesh from GMSH.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Reorder it with DMPlexPermute.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create necessary pre-processing arrays related to the mesh/problem.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create field(s) with multi-dofs.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Create residual vectors.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">Define a function to calculate the residual for each cell and, use SNES.</span></div><div><span style="font-family:"Trebuchet MS",sans-serif">As you can see, I'm not using FV or FE structures (most examples do). Now, I'm trying to implement this in parallel using a similar approach. However, I'm struggling to understand how to create corresponding vectors and how to obtain index sets for each processor. Is there a tutorial or paper that covers this topic?</span></div></div></blockquote><div><br></div><div>The intention was that there is enough information in the manual to do this.</div><div><br></div><div>Using PetscFE/PetscFV is not required. However, I strongly encourage you to use PetscSection. Without this, it would be incredibly hard to do what you want. Once the DM has a Section, it can do things like automatically create vectors and matrices for you. It can redistribute them, subset them, etc. The Section describes how dofs are assigned to pieces of the mesh (mesh points). This is in the manual, and there are a few examples that do it by hand.</div><div><br></div><div>So I suggest changing your code to use PetscSection, and then letting us know if things still do not work.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:"Trebuchet MS",sans-serif">Thank you.</span><br></div><div style="font-family:Arial,sans-serif;font-size:14px"><span style="font-family:"Trebuchet MS",sans-serif">Guer.</span></div><div style="font-family:Arial,sans-serif;font-size:14px"><br></div>
<div style="font-family:Arial,sans-serif;font-size:14px">
<div>
</div>
<div>
Sent with <a href="https://proton.me/" rel="noreferrer nofollow noopener" target="_blank">Proton Mail</a> secure email.
</div>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a rel="noreferrer nofollow noopener" href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div><div><br></div></div></div></div></div></div></div></div>
</blockquote><br>
</div>