<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 15, 2022, at 12:26 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Fri, Jul 15, 2022 at 2:12 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jul 15, 2022, at 11:58 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Fri, Jul 15, 2022 at 1:46 PM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><blockquote type="cite" class=""><div class="">On Jul 15, 2022, at 11:20 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Fri, Jul 15, 2022 at 11:01 AM Randall Mackie <<a href="mailto:rlmackie862@gmail.com" target="_blank" class="">rlmackie862@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">I am also interested in converting my DMDA code to DMPlex so that I can use OcTree grids.<div class=""><br class=""></div><div class="">Is there a simple example that would show how to do a box grid in DMPlex, or more information about how to convert a DMDA grid to DMPlex?</div></div></blockquote><div class=""><br class=""></div><div class="">Hi Randy,</div><div class=""><br class=""></div><div class="">Creating a box mesh is easy and can be done from the command line.</div><div class=""><br class=""></div><div class="">The hard part is usually converting the loop structure. Plex is setup to support FEM and FVM, which are both</div><div class="">cell-oriented. DMDA, on the other hand, tends to support a stencil of cells/vertices. Is this how your code looks?</div></div></div></div></blockquote><div class=""><br class=""></div>Hi Matt,</div><div class=""><br class=""></div><div class="">I figured the hard part was the loop structure.</div><div class=""><br class=""></div><div class="">Yes, my DMDA code is pretty standard and I have fields defined on block edges (it’s a staggered grid implementation, but written long before petsc had staggered grid capability) and I just loop over the DMDA grid points like:</div><div class=""><br class=""></div><div class="">do k=zs,ze</div><div class=""> do j=ys,ye</div><div class=""> do i=xs,xe</div><div class=""><br class=""></div><div class="">I’d be very interested to see what you can show us.</div></div></blockquote><div class=""><br class=""></div><div class="">What information do you need when computing an entry, for the cell and then for the face?</div></div></div></div></blockquote><div class=""><br class=""></div>I currently set up a 3D DMDA using a box stencil and a stencil width of 2.</div><div class="">The i,j,k coordinates refer both to the cell (where there is a physical value assigned) and to the 3 edges of the cell at the top SW corner.</div><div class="">For local computations, I need to be able to access the values up to +/- 2 grid points away.</div><div class=""><br class=""></div><div class="">I don’t really refer to the faces since that is implicitly included in the curl-curl formulation I am solving.</div><div class=""><br class=""></div><div class="">Is this what you are asking for?</div></div></blockquote><div class=""><br class=""></div><div class="">Yes. Unfortunately, this is hard. The topological definitions are all local, so even 1 layer of cells is awkward, but 2 layers</div><div class="">would be harder. With adaptivity, it gets harder still.</div><div class=""><br class=""></div><div class="">My approach, with Abhishek and Dave Salac, has been to preprocess all stencils and store them. Since p4est assumes</div><div class="">a Cartesian topology, it might be easier to directly use the p4est traversal. Toby might be better at explaining that.</div><div class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div>Hi Matt,</div><div><br class=""></div><div>So are you suggesting to use p4est outside of PETSc or is it possible to do the p4est traversal within PETSc?</div><div><br class=""></div><div>If I were to make it easy and say I just want to evaluate the typical curl-curl paddle for an OcTree grid (knowing that one has to deal with the hanging nodes in some way), would your PetscFD approach work or are there any examples that might give me a hint on how to proceed?</div><div><br class=""></div><div>It seems that one can represent OcTrees in PETSc using DMForest or DMPlex, but I’m not sure which is the right approach.</div><div><br class=""></div><div>Thanks, Randy</div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">Randy</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="">Thanks, Randy</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class="">I have a student working on a PetscFD which I could show you, but it is far from production.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="">Thanks, Randy<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Jun 21, 2022, at 10:57 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">(keep on the list, you will need Matt and Toby soon anyway).<div class=""><br class=""></div><div class="">So you want to add AMRex to your code.</div><div class=""><br class=""></div><div class="">I think the first thing that you want to do is move your DMDA code into a DMPLex code. You can create a "box" mesh and it is not hard.</div><div class="">Others like Matt can give advice on how to get started on that translation.</div><div class="">There is a simple step to create a DMForest (p4/8est) that Matt mentioned from the DMPlex .</div><div class=""><br class=""></div><div class="">Now at this point you can run your current SNES tests and get back to where you started, but AMR is easy now.</div><div class="">Or as easy as it gets.</div><div class=""><br class=""></div><div class="">As far as AMRex, well, it's not clear what AMRex does for you at this point. </div><div class="">You don't seem to have AMRex code that you want to reuse.</div><div class="">If there is some functionality that you need then we can talk about it or if you have some programmatic reason to use it (eg, they are paying you) then, again, we can talk about it.</div><div class=""><br class=""></div><div class="">PETSc/p4est and AMRex are similar with different strengths and design, and you could use both but that would complicate things.</div><div class=""><br class=""></div><div class="">Hope that helps,</div><div class="">Mark</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 21, 2022 at 1:18 PM Bernigaud Pierre <<a href="mailto:pierre.bernigaud@onera.fr" target="_blank" class="">pierre.bernigaud@onera.fr</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" class=""><p class="">Hello Mark, <br class="">
<br class="">
We have a working solver employing SNES, to which is attached a
DMDA to handle ghost cells / data sharing between processors for
flux evaluation (using DMGlobalToLocalBegin / DMGlobalToLocalEnd)
. We are considering to add an AMReX layer to the solver, but no
work has been done yet, as we are currently evaluating if it would
be feasible without too much trouble. <br class="">
<br class="">
Our main subject of concern would be to understand how to
interface correctly PETSc (SNES+DMDA) and AMRex, as AMRex also
appears to have his own methods for parallel data management.
Hence our inquiry for examples, just to get a feel for how it
would work out. <br class="">
</p><p class="">Best, <br class="">
<br class="">
Pierre <br class="">
</p>
<div class="">Le 21/06/2022 à 18:00, Mark Adams a
écrit :<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">Hi Bernigaud,
<div class=""><br class="">
</div>
<div class="">To be clear, you have SNES working with DMDA in AMRex, but
without adapting dynamically and you want to know what to do
next.</div>
<div class=""><br class="">
</div>
<div class="">Is that right?</div>
<div class=""><br class="">
</div>
<div class="">Mark</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jun 21, 2022 at 11:46
AM Bernigaud Pierre <<a href="mailto:pierre.bernigaud@onera.fr" target="_blank" class="">pierre.bernigaud@onera.fr</a>>
wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Greetings,<br class="">
<br class="">
I hope you are doing great.<br class="">
<br class="">
We are currently working on parallel solver employing PETSc
for the main <br class="">
numerical methods (GMRES, Newton-Krylov method). We would be
interested <br class="">
in combining the PETSc solvers with the AMR framework provided
by the <br class="">
library AMReX (<a href="https://amrex-codes.github.io/amrex/" rel="noreferrer" target="_blank" class="">https://amrex-codes.github.io/amrex/</a>).
I know that within <br class="">
the AMReX framework the KSP solvers provided by PETSc can be
used, but <br class="">
what about the SNES solvers? More specifically, we are using a
DMDA to <br class="">
manage parallel communications during the SNES calculations,
and I am <br class="">
wondering how it would behave in a context where the data
layout between <br class="">
processors is modified by the AMR code when refining the grid.<br class="">
<br class="">
Would you have any experience on this matter ? Is there any <br class="">
collaboration going on between PETsc and AMReX, or would you
know of a <br class="">
code using both of them?<br class="">
<br class="">
Respectfully,<br class="">
<br class="">
Pierre Bernigaud<br class="">
<br class="">
</blockquote>
</div>
</blockquote>
<div class="">-- <br class="">
<div class=""><span id="gmail-m_-606268870973855790gmail-m_-5723191737741463902gmail-m_-5097687481414530443cid:18187595b31c642f3251" class=""><jlgjjjnkhffoclfc.gif></span><span style="font-family:Arial,Helvetica,sans-serif;font-size:9pt;color:rgb(6,104,179)" class=""><strong class="">Pierre Bernigaud</strong><br class="">
Doctorant<br class="">
Département multi-physique pour l’énergétique<br class="">
Modélisation Propulsion Fusée<br class="">
Tél: +33 1 80 38 62 33
<p style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal" class=""><br class="">
ONERA - The French Aerospace Lab - Centre de Palaiseau<br class="">
6, Chemin de la Vauve aux Granges - 91123 PALAISEAU<br class="">
<span style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:bold;color:rgb(6,104,179)" class="">Coordonnées GPS :
48.715169, 2.232833</span><br class="">
<br class="">
Nous suivre sur : <a href="https://www.onera.fr/" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank" class="">www.onera.fr</a> | <a href="http://www.twitter.com/@onera_fr" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank" class="">Twitter</a> | <a href="http://www.linkedin.com/company/onera" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank" class=""> LinkedIn</a> | <a href="http://www.facebook.fr/thefrenchaerospacelab" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank" class="">Facebook</a> | <a href="https://www.instagram.com/onera_the_french_aerospace_lab" style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal;color:rgb(6,104,179)" target="_blank" class="">Instagram</a></p><p style="text-decoration:none;font-family:Arial,Helvetica,sans-serif;font-size:8pt;font-weight:normal" class=""><br class="">
Avertissement/disclaimer <a href="https://www.onera.fr/en/emails-terms" target="_blank" class=""><span style="border:0pt none;text-decoration:none;color:rgb(6,104,179);font-size:8pt" class="">https://www.onera.fr/en/emails-terms</span></a></p>
<span id="gmail-m_-606268870973855790gmail-m_-5723191737741463902gmail-m_-5097687481414530443cid:18187595b323b058cb32" class=""><dldmcfkmcojhebgb.png></span><p class=""><br class="">
</p>
</span></div>
</div>
</div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></body></html>