<div dir="ltr"><div dir="ltr">On Fri, Jul 15, 2022 at 2:12 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com">rlmackie862@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><br><div><br><blockquote type="cite"><div>On Jul 15, 2022, at 11:58 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr">On Fri, Jul 15, 2022 at 1:46 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank">rlmackie862@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><blockquote type="cite"><div>On Jul 15, 2022, at 11:20 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr">On Fri, Jul 15, 2022 at 11:01 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank">rlmackie862@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I am also interested in converting my DMDA code to DMPlex so that I can use OcTree grids.<div><br></div><div>Is there a simple example that would show how to do a box grid in DMPlex, or more information about how to convert a DMDA grid to DMPlex?</div></div></blockquote><div><br></div><div>Hi Randy,</div><div><br></div><div>Creating a box mesh is easy and can be done from the command line.</div><div><br></div><div>The hard part is usually converting the loop structure. Plex is setup to support FEM and FVM, which are both</div><div>cell-oriented. DMDA, on the other hand, tends to support a stencil of cells/vertices. Is this how your code looks?</div></div></div></div></blockquote><div><br></div>Hi Matt,</div><div><br></div><div>I figured the hard part was the loop structure.</div><div><br></div><div>Yes, my DMDA code is pretty standard and I have fields defined on block edges (it’s a staggered grid implementation, but written long before petsc had staggered grid capability) and I just loop over the DMDA grid points like:</div><div><br></div><div>do k=zs,ze</div><div> do j=ys,ye</div><div> do i=xs,xe</div><div><br></div><div>I’d be very interested to see what you can show us.</div></div></blockquote><div><br></div><div>What information do you need when computing an entry, for the cell and then for the face?</div></div></div></div></blockquote><div><br></div>I currently set up a 3D DMDA using a box stencil and a stencil width of 2.</div><div>The i,j,k coordinates refer both to the cell (where there is a physical value assigned) and to the 3 edges of the cell at the top SW corner.</div><div>For local computations, I need to be able to access the values up to +/- 2 grid points away.</div><div><br></div><div>I don’t really refer to the faces since that is implicitly included in the curl-curl formulation I am solving.</div><div><br></div><div>Is this what you are asking for?</div></div></blockquote><div><br></div><div>Yes. Unfortunately, this is hard. The topological definitions are all local, so even 1 layer of cells is awkward, but 2 layers</div><div>would be harder. With adaptivity, it gets harder still.</div><div><br></div><div>My approach, with Abhishek and Dave Salac, has been to preprocess all stencils and store them. Since p4est assumes</div><div>a Cartesian topology, it might be easier to directly use the p4est traversal. Toby might be better at explaining that.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div>Randy</div><div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Thanks, Randy</div><div><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_quote"><div><br></div><div>I have a student working on a PetscFD which I could show you, but it is far from production.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Thanks, Randy<br><div><br><blockquote type="cite"><div>On Jun 21, 2022, at 10:57 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">(keep on the list, you will need Matt and Toby soon anyway).<div><br></div><div>So you want to add AMRex to your code.</div><div><br></div><div>I think the first thing that you want to do is move your DMDA code into a DMPLex code. You can create a "box" mesh and it is not hard.</div><div>Others like Matt can give advice on how to get started on that translation.</div><div>There is a simple step to create a DMForest (p4/8est) that Matt mentioned from the DMPlex .</div><div><br></div><div>Now at this point you can run your current SNES tests and get back to where you started, but AMR is easy now.</div><div>Or as easy as it gets.</div><div><br></div><div>As far as AMRex, well, it's not clear what AMRex does for you at this point. </div><div>You don't seem to have AMRex code that you want to reuse.</div><div>If there is some functionality that you need then we can talk about it or if you have some programmatic reason to use it (eg, they are paying you) then, again, we can talk about it.</div><div><br></div><div>PETSc/p4est and AMRex are similar with different strengths and design, and you could use both but that would complicate things.</div><div><br></div><div>Hope that helps,</div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 21, 2022 at 1:18 PM Bernigaud Pierre <<a href="mailto:pierre.bernigaud@onera.fr" target="_blank">pierre.bernigaud@onera.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><p>Hello Mark, <br>
<br>
We have a working solver employing SNES, to which is attached a
DMDA to handle ghost cells / data sharing between processors for
flux evaluation (using DMGlobalToLocalBegin / DMGlobalToLocalEnd)
. We are considering to add an AMReX layer to the solver, but no
work has been done yet, as we are currently evaluating if it would
be feasible without too much trouble. <br>
<br>
Our main subject of concern would be to understand how to
interface correctly PETSc (SNES+DMDA) and AMRex, as AMRex also
appears to have his own methods for parallel data management.
Hence our inquiry for examples, just to get a feel for how it
would work out. <br>
</p><p>Best, <br>
<br>
Pierre <br>
</p>
<div>Le 21/06/2022 à 18:00, Mark Adams a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Bernigaud,
<div><br>
</div>
<div>To be clear, you have SNES working with DMDA in AMRex, but
without adapting dynamically and you want to know what to do
next.</div>
<div><br>
</div>
<div>Is that right?</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jun 21, 2022 at 11:46
AM Bernigaud Pierre <<a href="mailto:pierre.bernigaud@onera.fr" target="_blank">pierre.bernigaud@onera.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Greetings,<br>
<br>
I hope you are doing great.<br>
<br>
We are currently working on parallel solver employing PETSc
for the main <br>
numerical methods (GMRES, Newton-Krylov method). We would be
interested <br>
in combining the PETSc solvers with the AMR framework provided
by the <br>
library AMReX (<a href="https://amrex-codes.github.io/amrex/" rel="noreferrer" target="_blank">https://amrex-codes.github.io/amrex/</a>).
I know that within <br>
the AMReX framework the KSP solvers provided by PETSc can be
used, but <br>
what about the SNES solvers? More specifically, we are using a
DMDA to <br>
manage parallel communications during the SNES calculations,
and I am <br>
wondering how it would behave in a context where the data
layout between <br>
processors is modified by the AMR code when refining the grid.<br>
<br>
Would you have any experience on this matter ? Is there any <br>
collaboration going on between PETsc and AMReX, or would you
know of a <br>
code using both of them?<br>
<br>
Respectfully,<br>
<br>
Pierre Bernigaud<br>
<br>
</blockquote>
</div>
</blockquote>
<div>-- <br>
<div><span id="gmail-m_-606268870973855790gmail-m_-5723191737741463902gmail-m_-5097687481414530443cid:18187595b31c642f3251"><jlgjjjnkhffoclfc.gif></span><span style="font-family:Arial,Helvetica,sans-serif;font-size:9pt;color:rgb(6,104,179)"><strong>Pierre Bernigaud</strong><br>
Doctorant<br>
Département multi-physique pour l’énergétique<br>
Modélisation Propulsion Fusée<br>
Tél: +33 1 80 38 62 33
<p style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal"><br>
ONERA - The French Aerospace Lab - Centre de Palaiseau<br>
6, Chemin de la Vauve aux Granges - 91123 PALAISEAU<br>
<span style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:bold;color:rgb(6,104,179)">Coordonnées GPS :
48.715169, 2.232833</span><br>
<br>
Nous suivre sur : <a href="https://www.onera.fr/" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank">www.onera.fr</a> | <a href="http://www.twitter.com/@onera_fr" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank">Twitter</a> | <a href="http://www.linkedin.com/company/onera" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank"> LinkedIn</a> | <a href="http://www.facebook.fr/thefrenchaerospacelab" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank">Facebook</a> | <a href="https://www.instagram.com/onera_the_french_aerospace_lab" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank">Instagram</a></p><p style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal"><br>
Avertissement/disclaimer <a href="https://www.onera.fr/en/emails-terms" target="_blank"><span style="border:0pt none;text-decoration:none;color:rgb(6,104,179);font-size:8pt">https://www.onera.fr/en/emails-terms</span></a></p>
<span id="gmail-m_-606268870973855790gmail-m_-5723191737741463902gmail-m_-5097687481414530443cid:18187595b323b058cb32"><dldmcfkmcojhebgb.png></span><p><br>
</p>
</span></div>
</div>
</div>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>