<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">While I wait for a fix, I’m trying to setup DMPlex in our code by small increments.<div class="">I’m sorry for this very naive question but I have a hard time figuring out the DMPlex machinery for my specific needs.</div><div class="">What I need right now after calling DMCreateFromFile and DMPlexDistribute with an overlap of 1 is to get local DMPlexes (defined on PETSC_COMM_SELF, or at least with a local numbering of the elements) with for all elements, the (original) owner rank. Then, my plan is to convert these DMPlexes into the format needed by our code + a P_0 function defined locally (with overlap 1) with the value of the partitioning. That’d be a start.</div><div class="">Do I need to call DMPlexCreateTwoSidedProcessSF (some of the arguments are missing from the man page, <a href="https://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DMPLEX/DMPlexCreateTwoSidedProcessSF.html" class="">https://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DMPLEX/DMPlexCreateTwoSidedProcessSF.html</a>) with the SF returned by DMPlexDistribute? Or DMCreateSubDomainDM_Plex (which is not accessible from outside of PETSc I think?)? Or am I looking at the wrong place and overcomplicating all this?</div><div class=""><br class=""></div><div class="">Thank you in advance for your help,</div><div class="">Pierre</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 10 Feb 2020, at 4:30 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Mon, Feb 10, 2020 at 6:49 AM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" class="">pierre.jolivet@enseeiht.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 10 Feb 2020, at 1:08 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Sun, Feb 9, 2020 at 11:11 PM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank" class="">pierre.jolivet@enseeiht.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 10 Feb 2020, at 6:20 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Sun, Feb 9, 2020 at 3:23 PM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank" class="">pierre.jolivet@enseeiht.fr</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Hello,<br class="">I’ve a hard time answering the following DMPlex questions by just looking at some of the examples and manual.<br class="">Considering two DMPlex dm and dma, as in petsc/src/dm/impls/plex/examples/tests/ex19.c, I’d like to interpolate a simple P_1 FE function from dm to dma.<br class="">The DMCreateInterpolation call gives me:<br class="">[0]PETSC ERROR: Invalid argument<br class="">[0]PETSC ERROR: Number of fine indices 0 != 4 dual basis vecs<br class="">[…]<br class=""></blockquote><div class=""><br class=""></div><div class="">It looks like your fine grid has no discretization, since 0 is numFIndices from</div><div class=""><br class=""></div><div class=""> ierr = DMPlexGetClosureIndices(dmf, fsection, globalFSection, cell, &numFIndices, &findices, NULL);CHKERRQ(ierr);</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">[0]PETSC ERROR: #1 DMPlexComputeInterpolatorGeneral() line 2508 in petsc/src/dm/impls/plex/plexfem.c<br class="">[0]PETSC ERROR: #2 DMCreateInterpolation_Plex() line 7688 in petsc/src/dm/impls/plex/plex.c<br class="">[0]PETSC ERROR: #3 DMCreateInterpolation() line 1139 in petsc/src/dm/interface/dm.c<br class="">But the DMs look OK, don’t they, cf. below?<br class="">So I have three simple questions:<br class="">1) are all tests at the bottom of ex19.c broken because of PRAgMaTIc or because of DMPlex currently not supporting some operations? (I’m not using PRAgMaTIc to do mesh adaptation, so I was hoping to not run into an error)<br class=""></blockquote><div class=""><br class=""></div><div class="">I don't think its broken.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Oh, OK. Could you help me figure out what’s the problem then, e.g., with a slight (command line) variation of test #6, please?</div></div></div></blockquote><div class=""><br class=""></div><div class="">Sure. I am at SIAM this week, but as soon as I can I will get you the fix.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Excellent.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">$ cd src/dm/impls/plex/examples/tests</div><div class="">$ git diff ex19.c</div><div class="">$ make ex19</div><div class="">$ mpirun ./ex19 -dim 3 -nbrVerEdge 10 -dm_plex_separate_marker 0 -met 0 -hmin 0.1 -hmax 0.3 -init_dm_view -adapt_dm_view -do_L2 -petscspace_degree 1 -petscfe_default_quadrature_order 1 -dm_plex_hash_location</div><div class=""><div class="">[0]PETSC ERROR: Nonconforming object sizes</div><div class="">[0]PETSC ERROR: The section point closure size 0 != dual space dimension 4</div><div class="">[…]</div><div class="">[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 633 in src/dm/impls/plex/plexproject.c</div><div class="">[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 771 in src/dm/impls/plex/plexproject.c</div><div class="">[0]PETSC ERROR: #3 DMProjectFunctionLocal() line 7809 in src/dm/interface/dm.c</div><div class="">[0]PETSC ERROR: #4 DMProjectFunction() line 7766 in src/dm/interface/dm.c</div></div><div class=""><br class=""></div><div class="">If I comment the DMProjectFunction() call, I end up with the same error as in my first message in DMCreateInterpolation().</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">2) is DMCreateInterpolation + MatInterpolate the correct way of transferring one Vec from a DMPlex onto another?<br class=""></blockquote><div class=""><br class=""></div><div class="">That is the intent.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">3) if yes, by looking at the names of the arguments in DMPlexComputeInterpolatorGeneral, dmc and dmf, could you comment on the performance of this function for unrelated meshes, e.g., if both DMs are “fine” and not one coarse and the other fine (albeit non-nested), for simple P_k spaces.<br class=""></blockquote><div class=""><br class=""></div><div class="">In general, it is going to be horrible. Here is what it does: locate the fine quadrature points in the coarse grid and interpolate to them. This quadrature can have huge errors if it falls across multiple cells. This is why the nested version works perfectly, and also why Patrick Farrell and James Maddison have the Supermesh library, which makes a refinement of the mesh until the quadrature is accurate everywhere. That way they guarantee that at least the zeroth moment is preserved.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Two subquestions if I may:</div><div class="">1) are there any plans to have this integrated through an external package?</div></div></div></blockquote><div class=""><br class=""></div><div class="">In the absence of a plan, there is a hope. I would really like it to happen.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">2) if I understand you correctly, you answered about the numerical performance of the function. I can live with high interpolation errors if both meshes are “far" from each other. I was mostly interested in the parallel performance of the function.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Everything is purely local except for point location. Since it has never really been tested in this mode, I am sure the</div><div class="">scaling can be bad. I believe the default is to extrapolate if the point is not covered, which makes sense for mostly</div><div class="">matching meshes. There is parallel point location, but it is intended for a few points where we are sampling the solution,</div><div class="">rather than lots of points everywhere which you would get for non-matching meshes with different distributions. Could</div><div class="">you say what kind of situation you are trying to optimize for?</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">As a disclaimer, I’m not trying to optimize anything yet. Just assessing the potential benefits of using DMPlex.</div><div class="">We have many solvers using mesh adaptation, so I’ll give you just one specific example.</div><div class="">When doing shape optimization, e.g., <a href="http://www.cmap.polytechnique.fr/~florian.feppon/videos/04_08.webm" target="_blank" class="">http://www.cmap.polytechnique.fr/~florian.feppon/videos/04_08.webm</a>, we redistribute an adapted mesh and reinterpolate a level-set.</div></div></div></blockquote><div class=""><br class=""></div><div class="">Very cool.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">The only missing piece to have something fully distributed, now that the mesh adaptation may be done in parallel, is the interpolator, we basically do (1) in Dave’s email.</div><div class="">It would probably not extremely hard to implement this better, but if I can leverage DMPlex to take care of mesh (re)distribution + solution interpolation, I’d prefer invest time in hooking this into our code instead of implementing the wheel.</div></div></div></blockquote><div class=""><br class=""></div><div class="">I think you can. I should have some time to get your ex19 thing going.</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">Thanks,</div><div class="">Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="gmail_quote"><div class=""> <span class=""> </span>Thanks,</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><div class="">Thanks,</div><div class="">Pierre</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class=""><div class="gmail_quote"><div class=""> <span class=""> </span>Thanks,</div><div class=""><br class=""></div><div class=""> <span class=""> </span>Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Thanks in advance for your help,<br class="">Pierre<br class=""><br class="">$ mpirun -n 1 ./ex19 -msh in.msh -init_dm_view ::ascii_info -adapt_dm_view ::ascii_info -mat_view ::ascii_info -do_L2 -petscspace_degree 1<br class="">DM Object: DMinit 1 MPI processes<br class=""> <span class=""> </span>type: plex<br class="">DMinit in 3 dimensions:<br class=""> <span class=""> </span>0-cells: 1331<br class=""> <span class=""> </span>1-cells: 7930<br class=""> <span class=""> </span>2-cells: 12600<br class=""> <span class=""> </span>3-cells: 6000<br class="">Labels:<br class=""> <span class=""> </span>depth: 4 strata with value/size (0 (1331), 1 (7930), 2 (12600), 3 (6000))<br class=""> <span class=""> </span>Face Sets: 6 strata with value/size (4 (200), 1 (200), 5 (200), 2 (200), 3 (200), 6 (200))<br class=""> <span class=""> </span>Cell Sets: 1 strata with value/size (0 (6000))<br class="">DM Object: DMadapt (adapt_) 1 MPI processes<br class=""> <span class=""> </span>type: plex<br class="">DMadapt in 3 dimensions:<br class=""> <span class=""> </span>0-cells: 2905<br class=""> <span class=""> </span>1-cells: 18888<br class=""> <span class=""> </span>2-cells: 31368<br class=""> <span class=""> </span>3-cells: 15384<br class="">Labels:<br class=""> <span class=""> </span>depth: 4 strata with value/size (0 (2905), 1 (18888), 2 (31368), 3 (15384))<br class=""> <span class=""> </span>Face Sets: 6 strata with value/size (1 (200), 4 (200), 6 (200), 2 (200), 5 (200), 3 (200))<br class=""> <span class=""> </span>Cell Sets: 1 strata with value/size (0 (15384))<br class=""><br class=""><br class=""></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>