<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannernqzslrw { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonnqzslrw:hover, .pfptPrimaryButtonnqzslrw:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonnqzslrw:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
On Tue, Mar 12, 2024 at 6: 49 PM Adrian Croucher <a. croucher@ auckland. ac. nz> wrote: hi Matt, Thanks for that. I think that is a useful change, even though it doesn't actually fix my problem (i. e. my code still gets an error). I think
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannernqzslrw" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannernqzslrw" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannernqzslrw" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannernqzslrw" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div dir="ltr">On Tue, Mar 12, 2024 at 6:49 PM Adrian Croucher <<a href="mailto:a.croucher@auckland.ac.nz">a.croucher@auckland.ac.nz</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>hi Matt,</p>
<p>Thanks for that. I think that is a useful change, even though it
doesn't actually fix my problem (i.e. my code still gets an
error). I think now that although the error message I'm getting is
similar to the one that used to result from just a simple repeated
stratification, the underlying cause is different.</p>
<p>Looking more closely at the error message, it is saying that the
depth 2 stratum overlaps the depth 1 stratum, e.g.:</p>
<p>[0]PETSC ERROR: New depth 2 range [0,141) overlaps with depth 1
range [0,141)</p>
<p>That suggested to me that some of the strata in the
re-distributed DMPlex are not contiguous. I checked that by
looking at the cone and support sizes of the DMPlex points inside
DMPlexStratify_Topological_Private(). On one of the processes they
looked like this:</p>
<p>p: 0, cone: 7, support: 0 <br>
p: 1, cone: 7, support: 0 <br>
p: 2, cone: 7, support: 0 <br>
p: 3, cone: 6, support: 0 <br>
p: 4, cone: 6, support: 0 <br>
p: 5, cone: 6, support: 0 <br>
p: 6, cone: 2, support: 0 <br>
p: 7, cone: 1, support: 0 <br>
p: 8, cone: 0, support: 1 <br>
p: 9, cone: 0, support: 1 <br>
p: 10, cone: 1, support: 2 <br>
p: 11, cone: 1, support: 2 <br>
p: 12, cone: 1, support: 1 <br>
p: 13, cone: 1, support: 1 <br>
p: 14, cone: 2, support: 0 <br>
p: 15, cone: 2, support: 0 <br>
p: 16, cone: 1, support: 0 <br>
p: 17, cone: 1, support: 0 <br>
p: 18, cone: 0, support: 1 <br>
...</p>
<p>So here the cells (support 0) are not contiguous (and neither are
the other strata) - some of the dual-porosity cells (14 - 17) are
getting separated from the other cells (0 - 7). So I think that is
messing up the identification of stratum bounds, because it's
right, they do overlap. </p>
<p>Before redistribution, the strata are contiguous and
non-overlapping. So I think the redistribution is causing the
trouble.</p>
<p>When I first add the dual-porosity points to the DM I add them on
the end of each stratum, after interior points and partition ghost
points. That used to work, but perhaps something has changed so it
doesn't any more? Are there any assumptions made about the order
of storing interior points, partition ghost points and interior
ghost points?</p>
<p>Another possibility might be that it's getting confused by the
dual-porosity faces, edges, and nodes I'm adding which have
unusual cone/support sizes (e.g. faces with a cone of one edge,
which in turn have a cone of one node). Again this used to work ok
but perhaps something has changed so that it doesn't?</p></div></blockquote><div>Thanks for finding this. This was indeed rewritten because the old algorithm was too expensive at large sizes/parallelism. The function where this breaks in DMPlexStratifyMigrationSF() in plexdistribute.c. The problem is here:</div><div><br></div><div> <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexdistribute.c?ref_type=heads*L915__;Iw!!G_uCfscf7eWS!Yk85ASiGW0qV4yJakUojWQTr2JxXr5K44nxh2tKUmC3i-hL5fa7GwIu28UGfXCrmRT91IEVostCpYPLUl3bR$">https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/plexdistribute.c?ref_type=heads#L915</a></div><div><br></div><div>I treat DM_POLYTOPE_INTERIOR_GHOST as a cell no matter what. We need to assign the interior vertices, faces, and edges types that will give back the right dimension. Vertices are easy, they should just be type POINT. I can understand not wanting to use SEGMENT or QUAD for the interior edge and face because they do not have full cones. However, I think nothing will break (except explicit checks, which will not work anyway) if you give them types SEGMENT and QUAD. Is this easy to try?</div><div><br></div><div>I could also make a new type for you at each dimension if faking it does not work.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>
</p>
<p>- Adrian<br>
</p>
<div>On 9/03/24 7:15 am, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Okay, here are my first steps
<div class="gmail_quote">
<div><br>
</div>
<div>1. I fixed the problem with repeated stratification</div>
<div><br>
</div>
<div>knepley/fix-da-periodicity *$:/PETSc3/petsc/petsc-dev$
git diff<br>
diff --git a/src/dm/impls/plex/plex.c
b/src/dm/impls/plex/plex.c<br>
index 0ffc372ea08..9d27d308538 100644<br>
--- a/src/dm/impls/plex/plex.c<br>
+++ b/src/dm/impls/plex/plex.c<br>
@@ -4439,6 +4439,7 @@ PetscErrorCode DMPlexStratify(DM dm)<br>
PetscCall(PetscLogEventBegin(DMPLEX_Stratify, dm, 0, 0,
0));<br>
<br>
// Create depth label<br>
+ PetscCall(DMRemoveLabel(dm, "depth", NULL));<br>
PetscCall(DMCreateLabel(dm, "depth"));<br>
<br>
PetscCall(DMPlexGetDepthLabel(dm, &label));<br>
</div>
<div><br>
</div>
<div>I put it in an MR: <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7347__;!!G_uCfscf7eWS!Yk85ASiGW0qV4yJakUojWQTr2JxXr5K44nxh2tKUmC3i-hL5fa7GwIu28UGfXCrmRT91IEVostCpYFwOV8Q1$" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/7347</a></div>
<div><br>
</div>
<div>2. I wrote a small program to load the mesh, and then
restratify it. It is attached.</div>
<div><br>
</div>
<div>3. I ran stratification on this mesh, but I did not get
an error:</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<pre cols="72">--
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
Waipapa Taumata Rau / University of Auckland, New Zealand
email: <a href="mailto:a.croucher@auckland.ac.nz" target="_blank">a.croucher@auckland.ac.nz</a>
tel: +64 (0)9 923 4611</pre>
</div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!Yk85ASiGW0qV4yJakUojWQTr2JxXr5K44nxh2tKUmC3i-hL5fa7GwIu28UGfXCrmRT91IEVostCpYC-76Ov-$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>