<div dir="ltr"><div dir="ltr">On Mon, Feb 10, 2020 at 5:08 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com">dave.mayhem23@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Mon 10. Feb 2020 at 13:09, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Sun, Feb 9, 2020 at 11:11 PM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank">pierre.jolivet@enseeiht.fr</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><br><blockquote type="cite"><div>On 10 Feb 2020, at 6:20 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div><br><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr">On Sun, Feb 9, 2020 at 3:23 PM Pierre Jolivet <<a href="mailto:pierre.jolivet@enseeiht.fr" target="_blank">pierre.jolivet@enseeiht.fr</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>I’ve a hard time answering the following DMPlex questions by just looking at some of the examples and manual.<br>Considering two DMPlex dm and dma, as in petsc/src/dm/impls/plex/examples/tests/ex19.c, I’d like to interpolate a simple P_1 FE function from dm to dma.<br>The DMCreateInterpolation call gives me:<br>[0]PETSC ERROR: Invalid argument<br>[0]PETSC ERROR: Number of fine indices 0 != 4 dual basis vecs<br>[…]<br></blockquote><div><br></div><div>It looks like your fine grid has no discretization, since 0 is numFIndices from</div><div><br></div><div> ierr = DMPlexGetClosureIndices(dmf, fsection, globalFSection, cell, &numFIndices, &findices, NULL);CHKERRQ(ierr);</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[0]PETSC ERROR: #1 DMPlexComputeInterpolatorGeneral() line 2508 in petsc/src/dm/impls/plex/plexfem.c<br>[0]PETSC ERROR: #2 DMCreateInterpolation_Plex() line 7688 in petsc/src/dm/impls/plex/plex.c<br>[0]PETSC ERROR: #3 DMCreateInterpolation() line 1139 in petsc/src/dm/interface/dm.c<br>But the DMs look OK, don’t they, cf. below?<br>So I have three simple questions:<br>1) are all tests at the bottom of ex19.c broken because of PRAgMaTIc or because of DMPlex currently not supporting some operations? (I’m not using PRAgMaTIc to do mesh adaptation, so I was hoping to not run into an error)<br></blockquote><div><br></div><div>I don't think its broken.</div></div></div></div></blockquote><div><br></div><div>Oh, OK. Could you help me figure out what’s the problem then, e.g., with a slight (command line) variation of test #6, please?</div></div></div></blockquote><div><br></div><div>Sure. I am at SIAM this week, but as soon as I can I will get you the fix.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div>$ cd src/dm/impls/plex/examples/tests</div><div>$ git diff ex19.c</div><div>$ make ex19</div><div>$ mpirun ./ex19 -dim 3 -nbrVerEdge 10 -dm_plex_separate_marker 0 -met 0 -hmin 0.1 -hmax 0.3 -init_dm_view -adapt_dm_view -do_L2 -petscspace_degree 1 -petscfe_default_quadrature_order 1 -dm_plex_hash_location</div><div><div>[0]PETSC ERROR: Nonconforming object sizes</div><div>[0]PETSC ERROR: The section point closure size 0 != dual space dimension 4</div><div>[…]</div><div>[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 633 in src/dm/impls/plex/plexproject.c</div><div>[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 771 in src/dm/impls/plex/plexproject.c</div><div>[0]PETSC ERROR: #3 DMProjectFunctionLocal() line 7809 in src/dm/interface/dm.c</div><div>[0]PETSC ERROR: #4 DMProjectFunction() line 7766 in src/dm/interface/dm.c</div></div><div><br></div><div>If I comment the DMProjectFunction() call, I end up with the same error as in my first message in DMCreateInterpolation().</div><br><blockquote type="cite"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">2) is DMCreateInterpolation + MatInterpolate the correct way of transferring one Vec from a DMPlex onto another?<br></blockquote><div><br></div><div>That is the intent.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">3) if yes, by looking at the names of the arguments in DMPlexComputeInterpolatorGeneral, dmc and dmf, could you comment on the performance of this function for unrelated meshes, e.g., if both DMs are “fine” and not one coarse and the other fine (albeit non-nested), for simple P_k spaces.<br></blockquote><div><br></div><div>In general, it is going to be horrible. Here is what it does: locate the fine quadrature points in the coarse grid and interpolate to them. This quadrature can have huge errors if it falls across multiple cells. This is why the nested version works perfectly, and also why Patrick Farrell and James Maddison have the Supermesh library, which makes a refinement of the mesh until the quadrature is accurate everywhere. That way they guarantee that at least the zeroth moment is preserved.</div></div></div></div></blockquote><div><br></div><div>Two subquestions if I may:</div><div>1) are there any plans to have this integrated through an external package?</div></div></div></blockquote><div><br></div><div>In the absence of a plan, there is a hope. I would really like it to happen.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div>2) if I understand you correctly, you answered about the numerical performance of the function. I can live with high interpolation errors if both meshes are “far" from each other. I was mostly interested in the parallel performance of the function.</div></div></div></blockquote><div><br></div><div>Everything is purely local except for point location. Since it has never really been tested in this mode, I am sure the</div><div>scaling can be bad. I believe the default is to extrapolate if the point is not covered, which makes sense for mostly</div><div>matching meshes. There is parallel point location, but it is intended for a few points where we are sampling the solution,</div><div>rather than lots of points everywhere which you would get for non-matching meshes with different distributions. Could</div><div>you say what kind of situation you are trying to optimize for?</div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">Matt, probably clarifying the parallel point location algorithm is helpful.</div><div dir="auto"><br></div><div dir="auto">1/ Does it broadcast all off rank points to every rank?</div><div dir="auto">Or</div><div dir="auto">2/ Does it broadcast sub domain bounding boxes to every rank, and then scatter points to candidate owning ranks based on the bounding boxes containing off rank points?</div><div dir="auto"><br></div><div dir="auto">I recall the method (years ago) did what’s described in (1)</div></div></div></blockquote><div><br></div><div>I thought it was 2.</div><div><br></div><div>This is one of the things on the list when we scale up the plasmas physics code. My reading</div><div>says that a hierarchy of regular divisions is just as efficient for location as a structure for</div><div>irregular divisons, like a k-d tree, and we can keep the bottom part of the hierarchy everywhere.</div><div>Is that your feeling?</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><div>Thanks,</div><div>Pierre</div><br><blockquote type="cite"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div class="gmail_quote"><div> <span> </span>Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks in advance for your help,<br>Pierre<br><br>$ mpirun -n 1 ./ex19 -msh in.msh -init_dm_view ::ascii_info -adapt_dm_view ::ascii_info -mat_view ::ascii_info -do_L2 -petscspace_degree 1<br>DM Object: DMinit 1 MPI processes<br> <span> </span>type: plex<br>DMinit in 3 dimensions:<br> <span> </span>0-cells: 1331<br> <span> </span>1-cells: 7930<br> <span> </span>2-cells: 12600<br> <span> </span>3-cells: 6000<br>Labels:<br> <span> </span>depth: 4 strata with value/size (0 (1331), 1 (7930), 2 (12600), 3 (6000))<br> <span> </span>Face Sets: 6 strata with value/size (4 (200), 1 (200), 5 (200), 2 (200), 3 (200), 6 (200))<br> <span> </span>Cell Sets: 1 strata with value/size (0 (6000))<br>DM Object: DMadapt (adapt_) 1 MPI processes<br> <span> </span>type: plex<br>DMadapt in 3 dimensions:<br> <span> </span>0-cells: 2905<br> <span> </span>1-cells: 18888<br> <span> </span>2-cells: 31368<br> <span> </span>3-cells: 15384<br>Labels:<br> <span> </span>depth: 4 strata with value/size (0 (2905), 1 (18888), 2 (31368), 3 (15384))<br> <span> </span>Face Sets: 6 strata with value/size (1 (200), 4 (200), 6 (200), 2 (200), 5 (200), 3 (200))<br> <span> </span>Cell Sets: 1 strata with value/size (0 (15384))<br><br><br></blockquote></div><br clear="all"><div><br></div>--<span> </span><br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>