<div dir="ltr"><div dir="ltr">On Wed, Jul 20, 2022 at 6:25 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com">boraj1021@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for the comments.<br><br>But If the ordering before DMPlexDistribute is what you're looking for, the SF directly tells you the mapping. The leaves of the natural SF gives you the indices from your file. You just need to shift it by +1.<br><b>-> I do not agree. Below is my Natural SF with 2 proc, and if you see the attached snapshot from Gmsh with natural vertex index, the node index below y<=0.5 for proc=0 has the natural vertex index such as 22 (i.e., at x=0.25, y=0.25). But if the leaves of proc=0 are shifted by just +1, there is no such index 22. What do you think?</b></div></blockquote><div><br></div><div>The numbering below is the default PETSc numbering, namely cells, then vertices, then faces, then edges. For an even split, that would be 8 cells</div><div>per process, but it looks like you have an overlap, so there are more cells. You can see the numbering if you use</div><div><br></div><div> -dm_view ::ascii_info_detail</div><div><br></div><div>which would be useful to include here.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">PetscSF Object: 2 MPI processes<br> type: basic<br> [0] Number of roots=10, leaves=10, remote ranks=2<br> [0] 0 <- (0,0)<br> [0] 1 <- (1,0)<br> [0] 2 <- (1,1)<br> [0] 3 <- (0,1)<br> [0] 4 <- (0,2)<br> [0] 5 <- (1,2)<br> [0] 6 <- (1,3)<br> [0] 7 <- (1,4)<br> [0] 8 <- (1,5)<br> [0] 9 <- (1,6)<br> [1] Number of roots=15, leaves=15, remote ranks=2<br> [1] 0 <- (1,7)<br> [1] 1 <- (1,8)<br> [1] 2 <- (0,3)<br> [1] 3 <- (0,4)<br> [1] 4 <- (0,5)<br> [1] 5 <- (0,6)<br> [1] 6 <- (1,9)<br> [1] 7 <- (0,7)<br> [1] 8 <- (1,10)<br> [1] 9 <- (0,8)<br> [1] 10 <- (1,11)<br> [1] 11 <- (0,9)<br> [1] 12 <- (1,12)<br> [1] 13 <- (1,13)<br> [1] 14 <- (1,14)<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 20, 2022 at 4:41 PM Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Bora,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I agree with Matt. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I don't know what <span style="font-size:15px">
DMPlexCreateGmshFromFile() is doing in term of distribution then. But</span><b style="font-size:15px"> </b>If
the ordering before DMPlexDistribute is what you're looking for, the SF directly tells you the mapping. The leaves of the natural SF gives you the indices from your file. You just need to shift it by +1. If the leave indices (ilocal of PetscSFGetGraph) is
NULL, it means you have a contiguous storage. Each remote point (iremote <span style="background-color:rgb(255,255,255);display:inline">
of PetscSFGetGraph) </span>associated to a leave gives you the rank (iremote.rank) and the index (iremote.index) of your dof after DMPlexDistribute.</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Concerning the Petsc version, I am on main 3.17.3. In particular, DMPlexCreateGlobalToNatural was changed.
<span style="background-color:rgb(255,255,255);display:inline">
I am not familiar with previous version of DMPlexCreateGlobalToNaturalSF however so my help here is limited.</span> I think if you pull the latest changes on main you'll see an error when creating the natural SF: duplicate entries for the leaves. This is due
to the constraint dofs in you section. You can view your section to see if it effectively contains constrained dofs. I have a MR pending for fixing that issue. After the merge, the natural SF will map a Global Vec (after DMPlexDistribute in your case) to a
natural Vec (before DMPlexDistribute) including<span style="background-color:rgb(255,255,255);display:inline"> constraint dofs</span>.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">Best,
<div>Alexis</div>
</span></div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Alexis
Marboeuf</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Postdoctoral
fellow, Department of Mathematics & Statistics</span><br>
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Hamilton
Hall room 409B, McMaster University</span><br style="font-size:15px;text-align:start">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">1280
Main Street West, Hamilton, Ontario L8S 4K1, Canada</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">EMail:
<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Tel.
+1 (905) 525 9140 ext. 27031</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline"><span style="margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;text-align:left;background-color:rgb(255,255,255)"><span style="margin:0px;font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></span><br>
</span></div>
</div>
</div>
</div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> July 20, 2022 4:24 PM<br>
<b>To:</b> Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>><br>
<b>Cc:</b> Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>>; Blaise Bourdin <<a href="mailto:bourdin@mcmaster.ca" target="_blank">bourdin@mcmaster.ca</a>>; PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] DMPlex: a Mapping Array between Natural and Distributed Cell Index</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Wed, Jul 20, 2022 at 2:27 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I am not seeing DMPlexDistributeSetDefault(dm,PETSC_FALSE) after you create your DM from the GMSH file. Your DM is then automatically distributed and the natural ordering won't be a priori the one from your file. I don't know how Petsc distributes
a DM during the load step from a file.<br>
<b>-> PetscSFView() gives me the same Natural SF whether I have DMPlexDistributeSetDefault(dm,PETSC_FALSE) after DMPlexCreateGmshFromFile() or not. In addition, in my opinion if the Natural SF is not defined properly due to the distribution, I might see Null
from PetscSFView(). This might be previously checked with Matt thanks to the help.
</b><br>
<br>
a) You have here two differents ordering and distribution: the one given by Petsc after creating your DM from your file (which is distributed), i.e. before DMPlexDistribute and which is considered natural, and the one after DMPlexDistribute.<br>
<b>-> This ordering you mentioned (i.e., the ordering made after creating DM from mesh file but before distribution) is exactly what I want to have. Natural SF can show me its roots and leaves (i.e., graphs), but not the actual index written in the mesh file
(i.e., start from 1 to number of total vertex, for example). <br>
Adding DMPlexDistributeSetDefault(dm,PETSC_FALSE) does not really change my Natural SF. When you mention "sequantial" ordering, does it mean the same order written in the mesh file, instead of the graphs (i.e., roots and leaves)?</b><br>
<br>
b) I am not familiar with the overlap. From my understanding, even when you pass overlap = 0 to DMPlexDistribute, there is still an "overlap", i.e. points shared between procs. For your run on 2 procs, some dofs (5 dofs in your case) are on both procs when
it comes to local Vec but only one proc owes these dofs concerning global Vec. That's why your natural SF shows 5 dofs less on proc 0.<br>
<b>-> Thanks, makes sense. The number of stratum points I get from DMPlexGetDepthStratum() for vertex is the size of local Vec with 1dof on vertex.
<br>
</b><br>
Are you on the 'release' branch? On 'main', the creation of the natural SF raises an error (duplicate leave entries) when the section includes constraint dofs.<br>
<b>-> I am in the main branch of 3.17.0. It is unclear your comment to me, but I cannot see any duplicated leave entries in this case. There is 1dof on vertex.
<br>
</b><br>
<br>
Based on your comment, I might need to do; <br>
<b>1) Convert Natural SF's roots & leaves into the actual vertex index, which is written in the mesh file (i.e., from 1 to 25 in my case).</b></div>
</blockquote>
<div><br>
</div>
<div>If you define a scalar P1 field of one component, then the dof index will be the vertex index.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">2) DMPlexNaturalToGlobalBegin() and End() to pass the index from step 1) to global Vec</div>
</blockquote>
<div><br>
</div>
<div>I don't think you need this.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">3) DMGlobalToLocalBegin() and End() to pass the index to local Vec, finally.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I don't think you need this.</div>
<div><br>
</div>
<div>If you look at the SF, it tells you the mapping. However, you could get it in a round about way from a Vec. You could</div>
<div>put the index number at each position (e.g. put 5 at index 5). Then call GlobalToNatural(). This would tell you which</div>
<div>global index corresponds to each natural index.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Does this make sense? My questions so far are actually about step 1). As you see from the example code, I already defined Natural SF, which gives roots and leaves. Then how can I convert those graphs into the actual index, which is written in
the mesh file? <br>
<br>
Best<br>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jul 19, 2022 at 6:11 PM Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Bora,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for your file. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am not seeing DMPlexDistributeSetDefault(dm,PETSC_FALSE) after you create your DM from the GMSH file. Your DM is then automatically distributed and the natural ordering won't be a priori the one from your file. I don't know how Petsc distributes a DM during
the load step from a file.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
To address point after point your concerns:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<ul>
<li><span>a) You have here two differents ordering and distribution: the one given by Petsc after creating your DM from your file (which is distributed), i.e. before DMPlexDistribute and which is considered natural, and the one after <span style="background-color:rgb(255,255,255);display:inline">DMPlexDistribute.
Your natural SF maps between these two orderings. It means that you have 10 dofs (resp. 15 dofs) on proc 0 (resp. on proc 1) for both ordering. There is nothing wrong to me here. If you call "natural" the ordering from your file then you should call <span style="text-align:start;background-color:rgb(255,255,255);display:inline">DMPlexDistributeSetDefault(dm,PETSC_FALSE)<span> to
disable the automatic distribution by Petsc when you load from the file. It will give you a "sequential" ordering.</span></span></span></span></li><li><span><span style="background-color:rgb(255,255,255);display:inline">b) I am not familiar with the overlap. From my understanding, even when you pass overlap = 0 to DMPlexDistribute, there is still an "overlap", i.e. points shared between procs. For your
run on 2 procs, some dofs (5 dofs in your case) are on both procs when it comes to local Vec but only one proc owes these dofs concerning global Vec. That's why your natural SF shows 5 dofs less on proc 0.</span></span></li><li><span><span style="background-color:rgb(255,255,255);display:inline">c) When the storage is contiguous, PetscSFGetGraph gives NULL for ilocal.</span></span></li></ul>
<div>I have an additional question constraint dofs and the natural SF. Are you on the 'release' branch? On 'main', the creation of the natural SF raises an error (duplicate leave entries) when the section includes constraint dofs. I will submit a MR very soon
about that. But it will be merged to main.</div>
<div><br>
</div>
<div>Does this help?</div>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Best</div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Alexis Marboeuf</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Postdoctoral fellow, Department of Mathematics & Statistics</span><br>
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Hamilton Hall room 409B, McMaster University</span><br style="font-size:15px;text-align:start">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">EMail:
<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Tel. +1 (905) 525 9140 ext. 27031</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline"><span style="margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;text-align:left;background-color:rgb(255,255,255)"><span style="margin:0px;font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></span><br>
</span></div>
</div>
</div>
</div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>><br>
<b>Sent:</b> July 19, 2022 1:05 PM<br>
<b>To:</b> Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>><br>
<b>Cc:</b> Blaise Bourdin <<a href="mailto:bourdin@mcmaster.ca" target="_blank">bourdin@mcmaster.ca</a>>; Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] DMPlex: a Mapping Array between Natural and Distributed Cell Index</font>
<div> </div>
</div>
<div>
<div dir="ltr">Alexis, <br>
Thanks for the reply with notes. The sample code with 2d square mesh is attached here. I am not sure either there is something wrong with Natural SF or some limitations in my understanding. Hopefully, the latter one is the thing.<br>
<br>
The attached sample code follows the similar procedures that you & Matt have suggested;
<br>
- turn on DMSetUseNatural()<br>
- define section with 1dof on vertex before distribution of dm<br>
- manually distributing dm using DMPlexDistribute(), instead of relying on automatic distribution (e.g., DMSetFromOptions())<br>
- access graphs of Natural SF via PetscSFGetGraph()<br>
<br>
What I want to achieve; <br>
1) define my own local array "A" that maps local vertex index to natural order, for example,
<br>
A(local vertex index) = natural order of that local vertex. <br>
Here the "local vertex index" is obtained from DMPlexGetDepthStratum() with zero depth stratum, and mapped into 1 to number of vertex that each processor owns, for simplicity.<br>
Also, the "natural order" means the same indexing written in the mesh file.<br>
<b><u>2) Indeed, my inquiry is related to converting the vertex stratum from DMPlexGetDepthStratum() to the natural vertex index, which is written in the mesh file.
</u></b><br>
<br>
The problems, I have, are; <br>
a) I am a bit lost to interpret the meaning of values owned by the Natural SF. In the attached sample square mesh file, there are 25 vertex points. In the attached monitor file, Natural SF owns 10 roots & leaves for proc=0, and 15 roots & leaves for proc=1.
In this case, how can I map these "local" and "remote" arrays into the natural vertex index?
<br>
b) related to item a), DMPlexGetDepthStratum() is saying each processor owns 15 vertex points if np=2 without overlap. But Natural SF gives only 10 roots & leaves for proc=0. Is there anything wrong? or did I misunderstand something?
<br>
c) iroots pointer from PetscSFGetGraph() seems to have garbage data. By any chance, is it related to Fortran?<br>
<br>
Best<br>
</div>
<br>
<div>
<div dir="ltr">On Tue, Jul 19, 2022 at 11:01 AM Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am unfortunately unable to access the attached code.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Bora: have you defined constraint dofs in your section? In that case the natural SF is wrong, and you can notice duplicate entries for the roots if I remember correctly. I'll do a MR soon about that.</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The natural ordering and distribution is the one implemented by the DM before you call DMPlexDistribute. If your DM is automatically distributed when you load it from your GMSH file thanks to DMPlexCreateFromFile for example, you're not going to find your file
ordering in the natural SF. You may call DMPlexDistributeSetDefault(dm,PETSC_FALE) to disable the automatic distribution.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Best</div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716x_gmail-m_-4626791976940165187Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Alexis Marboeuf</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Postdoctoral fellow, Department of Mathematics & Statistics</span><br>
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Hamilton Hall room 409B, McMaster University</span><br style="font-size:15px;text-align:start">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">EMail:
<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a></span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">Tel. +1 (905) 525 9140 ext. 27031</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline"><span style="margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;text-align:left;background-color:rgb(255,255,255)"><span style="margin:0px;font-size:15px;text-align:start;background-color:rgb(255,255,255);display:inline">-------------------------------------------------------------------</span></span><br>
</span></div>
</div>
</div>
</div>
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716x_gmail-m_-4626791976940165187appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716x_gmail-m_-4626791976940165187divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Blaise Bourdin <<a href="mailto:bourdin@mcmaster.ca" target="_blank">bourdin@mcmaster.ca</a>><br>
<b>Sent:</b> July 18, 2022 8:52 PM<br>
<b>To:</b> Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>; Alexis Marboeuf <<a href="mailto:marboeua@mcmaster.ca" target="_blank">marboeua@mcmaster.ca</a>><br>
<b>Subject:</b> Re: [petsc-users] DMPlex: a Mapping Array between Natural and Distributed Cell Index</font>
<div> </div>
</div>
<div>Alexis noticed a problem with the natural SF when constraints are defined. He has a MR coming.
<div>@Alexis: Could it be that what Bora sees is exactly what you fixed?</div>
<div><br>
</div>
<div>Blaise<br>
<div><br>
<blockquote type="cite">
<div>On Jul 18, 2022, at 8:35 PM, Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Thank you for the reply.
<div><br>
<div>I am actually putting 1dof on the vertex points.
<div>In the attached sample code, from line 63 to line 99, 1dof on all vertex points is defined for the section. But my problem is, if I print out Natural SF using the viewer, I only have 10 vertex points, instead of 15. Here 15 is the size of the global vector
defined by the aforementioned section. The working example code was attached to the previous email with a small grid. </div>
</div>
<div><br>
</div>
<div>Best</div>
</div>
</div>
<br>
<div>
<div dir="ltr">On Mon, Jul 18, 2022 at 6:50 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, Jul 18, 2022 at 4:30 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Actually, it seems that my problem is that I do not properly understand what Natural SF means.
<br>
<br>
What I want to achieve is to extract natural vertex index from Natural SF, and make sure this natural vertex index is the same with what is written in a mesh file from Gmsh. The natural vertex index defined by Gmsh is attached as a snapshot with sample code
and mesh. I want to reproduce this indexing based on the Natural SF that Petsc defines.<br>
<br>
I am struggling with that because the number of entries defined in the Natural SF that I can see through PetscSFView() is different from the number of vertex points that each processor actually owns. For example, with two processors and with the attached "square.msh",
each processor might have 15 vertex points if I configure Petsc with ParMetis. However, the defined Natural SF for proc=0 only has 10 roots. I expected, for each processor, the Natural SF would show me 15 entries through PetscSFView(), if I define 1 degree
of freedom on vertex when the Petsc Section is defined. <br>
<br>
When Petsc defines the Natural SF, are there any omitted components since they are trivial to save?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>The naturalSF is mapping the field defined by the Section, not the mesh. You can get the effect you are asking for by putting one</div>
<div>degree of freedom (dof) on every vertex for the Section. How are you defining your Section now?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Best<br>
</div>
<br>
<div>
<div dir="ltr">On Mon, Jul 18, 2022 at 3:50 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Mon, Jul 18, 2022 at 12:14 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Thank you for the corrections. It works. <br>
<br>
However, I still have a problem. <br>
I tested zero stratum for DMGetStratumIS() with the default depth label, since the graph of vertex is needed. Then, PetscSFGetGraph() from the Natural SF is crashed. I checked that pBcCompIS contains proper set of integers via ISView().
<br>
<br>
On the other hand, it works okay if DMGetStratumIS() is with dim stratum, which is for cell stratum.
<br>
The problem is that if the dim stratum is used, the number of entry in ileaves(i)% pointer does not match with the actual number of vertex that each processor has. That is why I tried to put the zero stratum on DMGetStratumIS(), instead of the stratum for cell
(i.e., "dim") to see if any changes on the graph. Can I get comments?<br>
</div>
</blockquote>
<div><br>
</div>
<div>I cannot understand your question. I am not sure what you are using the set of vertices or cell for.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Best<br>
</div>
<br>
<div>
<div dir="ltr">On Sun, Jul 17, 2022 at 9:59 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jul 15, 2022 at 7:05 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I found that iroots() and ileaves() have size of nleaves and tested through the attached sample code with the grid previously shared. Then I get the results written in the attached monitor file.
<br>
<br>
It seems ileaves(i)%rank and ileaves(i)%index at line 127 of the attached code have garbage values, different from displayed values by PetscSFView() at line 113.
<br>
<br>
It is tough to get it why the garbage values are returned from PetscSFGetGraph(). Any comments will be very appreciated.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Unfortunately, Fortran is not very good at reporting declaration errors. The problem is that you did not include or use the Vec module. I have done this and your example runs for me. I have included the modified code.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Best<br>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jul 15, 2022 at 3:53 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jul 15, 2022 at 2:25 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Thank you for the comments. Updated sample code is attached here; DMPlexDistribute() is explicitly called and by defining a section before mesh distribution, a Natural SF was able to be defined as found from PetscSFView().
<br>
<br>
However, I am still struggling to call PetscSFGetGraph() <a href="https://petsc.org/main/docs/manualpages/PetscSF/PetscSFGetGraph/" target="_blank">
https://petsc.org/main/docs/manualpages/PetscSF/PetscSFGetGraph/</a><br>
due to data type definition of ilocal and iremote. What is the proper size allocation for those two variables? Does this size for the number of processors? or the number of vertex?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Since we need to pass back arrays, you need to pass us in F90 pointers. Here is an example of doing such a thing:</div>
<div><br>
</div>
<div> <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tutorials/ex1f.F90#L94" target="_blank">https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tutorials/ex1f.F90#L94</a></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Best<br>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jul 15, 2022 at 9:09 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jul 15, 2022 at 8:46 AM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Okay, I got it what's happening. First, this very bright functionality of petsc (the natural-global converting) needs to be documented in a better way for sure. Currently, it is very difficult to use/follow this features as an external user. Hope this
will move forward in a better way. </div>
<div><br>
</div>
<div>Then next question is if I load/distribute mesh just like I am doing right now shown in the sample code, can I assume that my code does not create natural sf during the distribution(always)? In other words, can I always get the natural order of each node
by simply stacking the preceding processor's number of node? For example, for proc=0, natural node ID might be just 1 to nnode_proc_0, </div>
<div>for proc=1, it might be {nnode_proc_0 + 1 to nnode_proc_0 + nnode_proc_1} and so on. </div>
<div><br>
</div>
<div>Does that always make sense in this case? </div>
</div>
</blockquote>
<div><br>
</div>
<div>No, but if you call DMPlexDistribute() yourself, rather than having it called automatically by DMSetFromOptions(), you can</div>
<div>preserve the mapping:</div>
<div><br>
</div>
<div> <a href="https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexDistribute/" target="_blank">
https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexDistribute/</a></div>
<div><br>
</div>
<div>The SF returned maps the original point distribution, which is in the same order as the file, to the redistributed points, which are determined</div>
<div>by the mesh partitioner.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt </div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Best</div>
<div dir="ltr"><br>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jul 15, 2022 at 8:07 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Fri, Jul 15, 2022 at 7:17 AM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>A sample code for loading dm and declaring Natural SF from a sphere mesh is attached here. PetscSFView() returns NULL from sf_nat. </div>
</div>
</blockquote>
<div><br>
</div>
<div>The Global-to-Natural mapping relates global dofs to natural dofs. Thus, in order to compute this mapping the DM has to know</div>
<div>about the dof layout before distribution. This means you need to setup a local section before distributing, as we do for example in</div>
<div>Plex test ex15. This makes things more complicated since everything cannot be packaged up into DMSetFromOptions(), but we need</div>
<div>user input so I do not see a simpler way to do things.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Best</div>
<br>
<div>
<div dir="ltr">On Fri, Jul 15, 2022 at 6:39 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Thu, Jul 14, 2022 at 8:25 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Okay, I checked it and you are correct. In my case, simply, natural node index can be identified by stacking all the preceding processor's numbers of nodes for a particular processor, which is good due to simplicity. However, one serious question is why
this is happening in my code? In other words, why the natural SF is not created during the mesh distribution? My code wants to have consistency in dealing with this natural indexing for several different kinds of mesh files. So there is a necessity to guarantee
of consistency in this weird behavior.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I can't tell what is going on in your code unless I can run it. Do you have a simple example?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div></div>
<div>Best,</div>
<div><br>
</div>
<div>
<div dir="ltr">On Thu, Jul 14, 2022 at 6:43 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Thu, Jul 14, 2022 at 5:47 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Thank you for the comments. I have these errors when I call PetscSFView() after DMGetNaturalSF()<br>
<br>
[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
[2]PETSC ERROR: Null argument, when expecting valid pointer<br>
[2]PETSC ERROR: Null Pointer: Parameter # 1<br>
[2]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<br>
[2]PETSC ERROR: Petsc Release Version 3.17.0, unknown <br>
[2]PETSC ERROR: #1 PetscSFView() at <br>
[2]PETSC ERROR: #2 User provided function() at User file:0<br>
Abort(85) on node 2 (rank 0 in comm 16): application called MPI_Abort(MPI_COMM_SELF, 85) - process 0<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Clearly NULL was returned, which means no natural SF was created.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Below is the structure to load a mesh file from gmsh; <br>
<br>
call DMCreate(PETSC_COMM_WORLD, dm, ierr);CHKERRA(ierr)<br>
call DMSetType(dm, plex, ierr);CHKERRA(ierr)<br>
call DMSetUseNatural(dm, PETSC_TRUE, ierr);CHKERRA(ierr)<br>
call DMSetFromOptions(dm, ierr);CHKERRA(ierr)<br>
call DMGetNaturalSF(dm, sf_nat, ierr);CHKERRA(ierr)<br>
call PetscSFView(sf_nat, PETSC_VIEWER_STDOUT_WORLD, ierr);CHKERRA(ierr)<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>The natural SF is created during mesh distribution. That has not happened here. This means that</div>
<div>the order of cells is identical to the file it was read from.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Best<br>
</div>
<br>
<div>
<div dir="ltr">On Thu, Jul 14, 2022 at 10:49 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">On Wed, Jul 13, 2022 at 10:17 PM Bora Jeong <<a href="mailto:boraj1021@gmail.com" target="_blank">boraj1021@gmail.com</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Dear petsc team,
<div> <br>
I am a user of DMPlex for a finite volume code and there is a necessity to know global index of each cell. Here the global index means the indexing that can be found from a mesh file itself without distribution over processors. It seems petsc community denotes
this indexing term as "natural". <br>
<br>
What I want to do is to create a local array (not petsc vector but just an array variable in the program) to map distributed cell ID to natual cell ID, for example, an array "A";
<br>
A(distributed_node_ID) = natural_node_ID<br>
<br>
There are some petsc functions to support mapping between global and natural vectors. However, I just need to define the array "A" as above example. To achieve this, what is a proper/smart way? In other words, how can I extract the natural_cell_ID from a distributed
local_cell_ID? <br>
<br>
I turned on DMSetUseNatural(DM, PETSC_TRUE) before distribution, but after this, defining all the required section and star forest objects to get natural and global vectors seems not that direct way for my purpose, which is just to extract the above mapping
array "A". Can I get any comments about it?<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>There is only one thing created, the sfNatural PetscSF object, which you can get with DMGetNaturalSF(). The roots of this SF are</div>
<div>the global numbers of dofs stored in PETSc vectors, and the leaves are natural numbers for these dofs. Thus, when we map global</div>
<div>vectors to natural vectors in DMPlexGlobalToNaturalBegin/End(), we call PetscSFBcastBegin/End(). Mapping natural to global we call</div>
<div>PetscSFReduceBegin/End(). You could pull the information out of the SF using PetscSFGetGraph() if you want.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Regards<br>
Mo<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<span id="gmail-m_-4198625806465656293gmail-m_-2382298079429289349x_gmail-m_4083276322460157814gmail-m_-2301289079634126716x_gmail-m_-4626791976940165187x_cid:f_l5rfzgvx0"><test.F90></span></div>
</blockquote>
</div>
<br>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div>— <br>
Tier 1 Canada Research Chair in Mathematical and Computational Aspects of Solid Mechanics</div>
<div>Professor, Department of Mathematics & Statistics<br>
Hamilton Hall room 409A, McMaster University<br>
1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada <br>
<a href="https://www.math.mcmaster.ca/bourdin" target="_blank">https://www.math.mcmaster.ca/bourdin</a> | +1 (905) 525 9140 ext. 27243</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>