<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Segoe UI Symbol";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1, li.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1, div.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241s1
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241s1;}
span.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3;}
p.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1, li.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1, div.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1;}
span.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space;}
span.gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2
{mso-style-name:gmail-m_-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <knepley@gmail.com><br><b>Date: </b>Wednesday, April 8, 2020 at 5:32 PM<br><b>To: </b>Danyang Su <danyang.su@gmail.com><br><b>Cc: </b>PETSc <petsc-users@mcs.anl.gov><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>On Wed, Apr 8, 2020 at 8:18 PM Danyang Su <<a href="mailto:danyang.su@gmail.com">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 4:50 PM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 7:47 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 4:41 PM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 5:52 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matt, <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1>I am one step closer now. When run the ex1 code with ‘<span class=gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241s1>-interpolate</span>’, the partition is good, without it, it’s weird.<o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Crap! That did not even occur to me. Yes, the dual graph construction will not work for uninterpolated wedges.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>So, do you really need an uninterpolated mesh? If so, I can put it on the buglist.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>For Prism mesh, I am afraid so. For 2D triangle mesh and 3D tetra mesh, the partition is pretty good without interpolate. That’s why I didn’t have problem for all my previous simulations using the other cell types.<o:p></o:p></p></div></div></div></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What I mean is, are you avoiding interpolating the mesh for memory? The amount of memory is usually small compared to<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>fields on the mesh.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>No, not because of memory consumption problem. When the code was first written several years ago, I just put interpolate = false there. Now after setting interpolate = true, I need to update the code in setting cell-node index (array cell). The following code does not work anymore when interpolate = true. There is some code that is not well written and it needs to be improved. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!c add local to global cell id mapping</span></i><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>do</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> ipoint </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>0</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>, istart</span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>-1</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> icell </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> ipoint </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>+</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>1</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMPlexGetCone</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(dmda_flow</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>%</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>da,ipoint,cone,ierr)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>do</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> ivtex </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>1</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>, num_nodes_per_cell</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> cell_node_idx(:,ipoint</span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>+1</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>) </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> cone </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> istart </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>+</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>1</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>end do</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMPlexRestoreCone</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(dmda_flow</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>%</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>da,ipoint,cone,ierr)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>end do</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>My F90 is a bit shaky, but I think you want<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> PetscInt, pointer :: nClosure(:)<br><br> do ipoint = 0, istart-1<br> icell = ipoint + 1<br> call DMPlexGetClosure(dmda_flow%da,ipoint,cone,ierr);CHKERRQ(ierr)<br> call DMPlexGetTransitiveClosure(dmda_flow%da,ipoint,PETSC_TRUE,nClosure,ierr);CHKERRQ(ierr)<br> ivtex = 0<br> do icl = 0,len(nClosure)-1,2<br> if ((nClosure(icl) >= vStart) .and. (nClosure(icl) < vEnd)) then<br> cell_node_idx(ivtex,ipoint+1) = nClosure(icl) - istart + 1<br> ivtex = ivtex + 1<br> end if<br> end do<br> call DMPlexRestoreTransitiveClosure(dmda_flow%da,ipoint,PETSC_TRUE,nClosure,ierr);CHKERRQ(ierr)<br> end do<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Basically, you use the closure, and filter out everything that is not a vertex.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks, Matt. Would you mind give some tips on the following code as well. The last line of following section (<span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>num_cells_loc </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_cells</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>+</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>num_nodes</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>nleaves</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>num_nodes_loc</span>) does not work either when <i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>interpolate == true.</span></i><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!c get local mesh DM</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMGetCoordinatesLocal</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(dmda_flow</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>%</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>da,gc,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p> </o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMGetCoordinateDM</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(dmda_flow</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>%</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>da,cda,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p> </o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMGetSection</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(cda,cs,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p> </o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>PetscSectionGetChart</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(cs,istart,iend,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p> </o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!c Calculate number of nodes/cells with ghost nodes/cells for each processor</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_nodes </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> iend</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>istart<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_cells </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> istart<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> <o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!c Calculate local number of nodes without ghost nodes</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_nodes_loc </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>0</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>do</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> ipoint </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> istart, iend</span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>-1</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMPlexGetPointGlobal</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(cda,ipoint,pstart,pend,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>if</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> (pend </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>0</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>) </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>then</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_nodes_loc </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_nodes_loc </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>+</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#D19A66'>1</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>end if</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>end do</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> <o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!c Calculate number of cells without ghost cells for each processor</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>DMGetPointSF</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(dmda_flow</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>%</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>da,sf,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> <o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>call</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><span style='font-size:10.5pt;font-family:Menlo;color:#61AFEF'>PetscSFGetGraph</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>(sf,nroots,nleaves,gmine,gremote,ierr)<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> CHKERRQ(ierr) <o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> <o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!!!!!This calculation is correct when interpolate == false!!!!!</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> </span><i><span style='font-size:10.5pt;font-family:Menlo;color:#5C6370'>!!!!! but incorrect when interpolate == true !!!!!</span></i><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_cells_loc </span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>=</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'> num_cells</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>+</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>num_nodes</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>nleaves</span><span style='font-size:10.5pt;font-family:Menlo;color:#C678DD'>-</span><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'>num_nodes_loc<o:p></o:p></span></p><p class=MsoNormal style='line-height:15.75pt;background:#282C34'><span style='font-size:10.5pt;font-family:Menlo;color:#ABB2BF'><o:p> </o:p></span></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Danyang<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> Matt<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1>Thanks,<o:p></o:p></p><p class=gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1> <o:p></o:p></p><p class=gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241p1>Danyang<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 2:12 PM<br><b>To: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matt,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Attached is another prism mesh using 8 processors. The partition of the lower mesh does not looks good. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=1187 height=1033 style='width:12.3645in;height:10.7604in' id="gmail-m_-3435706974861814781gmail-m_-1037530915984171460gmail-m_-7031366058158567241Picture_x0020_5" src="cid:image001.png@01D60DCD.996B4850"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 1:50 PM<br><b>To: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matt,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Here is what I get using ex1c with stencil 0. There is no change in the source code. I just compile and run the code in different ways. By using ‘make -f ./gmakefile ….’, it works as expected. However, by using ‘make ex1’ and then run the code using ‘mpiexec -n …’, the partition does not looks good. My code has the same problem as this one if I use prism mesh.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I just wonder what makes this difference, even without overlap.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><img border=0 width=642 height=802 style='width:6.6875in;height:8.3541in' id="gmail-m_-3435706974861814781gmail-m_-1037530915984171460gmail-m_-7031366058158567241Picture_x0020_4" src="cid:image002.png@01D60DCD.996B4850"><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 1:32 PM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 4:26 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 12:50 PM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 3:22 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matt,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Here is something pretty interesting. I modified ex1.c file with output of number of nodes and cells (as shown below) . And I also changed the stencil size to 1.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> /* get coordinates and section */<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> ierr = DMGetCoordinatesLocal(*dm,&gc);CHKERRQ(ierr);<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> ierr = DMGetCoordinateDM(*dm,&cda);CHKERRQ(ierr);<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> ierr = DMGetSection(cda,&cs);CHKERRQ(ierr);<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> ierr = PetscSectionGetChart(cs,&istart,&iend);CHKERRQ(ierr);<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> num_nodes = iend-istart;<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> num_cells = istart;<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> /* Output rank and processor information */<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:9.75pt'>printf("rank %d: of nprcs: %d, num_nodes %d, num_cess %d\n", rank, size, num_nodes, num_cells); <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>If I compile the code using ‘make ex1’ and then run the test using ‘<span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">mpiexec -n 2 ./ex1 -filename basin2layer.exo</span>’, I get the same problem as the modified ex1f90 code I sent.<o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1"><b><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"><b><span style='color:#39C026'> </span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2"><b>tests</b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3"> mpiexec -n 2 ./ex1 -filename basin2layer.exo</span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"> </span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 1: of nprcs: 2, num_nodes 699, num_cess 824</span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 0: of nprcs: 2, num_nodes 699, num_cess 824</span><o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Ah, I was not looking closely. You are asking for a cell overlap of 1 in the partition. That is why these numbers sum to more than<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>the total in the mesh. Do you want a cell overlap of 1?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Yes, I need cell overlap of 1 in some circumstance. The mesh has two layers of cells with 412 cells per layer and three layers of nodes with 233 nodes per layer. The number of cells looks good to me. I am confused why the same code generates pretty different partition. If I set the stencil to 0, I get following results. The first method looks good and the second one is not a good choice, with much more number of ghost nodes.<o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1"><b><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"><b><span style='color:#39C026'> </span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2"><b>petsc-3.13.0</b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3"> make -f ./gmakefile test globsearch="dm_impls_plex_tests-ex1_cylinder" EXTRA_OPTIONS="-filename ./basin2layer.exo -dm_view hdf5:$PWD/mesh.h5 -dm_partition_view" NP=2</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'># > rank 1: of nprcs: 2, num_nodes 354, num_cess 392<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'># > rank 0: of nprcs: 2, num_nodes 384, num_cess 432<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span> tests mpiexec -n 2 ./ex1 -filename basin2layer.exo<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>rank 0: of nprcs: 2, num_nodes 466, num_cess 412<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>rank 1: of nprcs: 2, num_nodes 466, num_cess 412<o:p></o:p></p></div></div></div></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I think this might just be a confusion over interpretation. Here is how partitioning works:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> 1) We partition the mesh cells using ParMetis, Chaco, etc.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> 2) We move those cells (and closures) to the correct processes<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> 3) If you ask for overlap, we mark a layer of adjacent cells on remote processes and move them to each process<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The original partitions are the same, Then we add extra cells, and their closures, to each partition. This is what you are asking for.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>You would get the same answer with GMsh if it gave you an overlap region.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1"><b><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"><b><span style='color:#39C026'> </span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2"><b>tests</b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3"> mpiexec -n 4 ./ex1 -filename basin2layer.exo</span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 1: of nprcs: 4, num_nodes 432, num_cess 486</span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 0: of nprcs: 4, num_nodes 405, num_cess 448</span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 2: of nprcs: 4, num_nodes 411, num_cess 464</span><o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3">rank 3: of nprcs: 4, num_nodes 420, num_cess 466</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>However, if I compile and run the code using the script you shared, I get reasonable results.<o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1"><b><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"><b><span style='color:#39C026'> </span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2"><b>petsc-3.13.0</b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3"> make -f ./gmakefile test globsearch="dm_impls_plex_tests-ex1_cylinder" EXTRA_OPTIONS="-filename ./basin2layer.exo -dm_view hdf5:$PWD/mesh.h5 -dm_partition_view" NP=2</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 0: of nprcs: 2, num_nodes 429, num_cess 484</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 1: of nprcs: 2, num_nodes 402, num_cess 446</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125p1"><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s1"><b><span style='font-family:"Segoe UI Symbol",sans-serif'>➜</span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125apple-converted-space"><b><span style='color:#39C026'> </span></b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s2"><b>petsc-3.13.0</b></span><span class="gmail-m-3435706974861814781gmail-m-1037530915984171460gmail-m-7031366058158567241gmail-m1766518872680001549gmail-m-6754249856326310125s3"> make -f ./gmakefile test globsearch="dm_impls_plex_tests-ex1_cylinder" EXTRA_OPTIONS="-filename ./basin2layer.exo -dm_view hdf5:$PWD/mesh.h5 -dm_partition_view" NP=4</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 1: of nprcs: 4, num_nodes 246, num_cess 260</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 2: of nprcs: 4, num_nodes 264, num_cess 274</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 3: of nprcs: 4, num_nodes 264, num_cess 280</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:8.5pt;font-family:Menlo;color:black'># > rank 0: of nprcs: 4, num_nodes 273, num_cess 284</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:9.75pt'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Is there some difference in compiling or runtime options that cause the difference? Would you please check if you can reproduce the same problem using the modified ex1.c?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Danyang<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 9:37 AM<br><b>To: </b>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 9:20 AM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 12:13 PM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Date: </b>Wednesday, April 8, 2020 at 6:45 AM<br><b>To: </b>Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>><br><b>Cc: </b>PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject: </b>Re: [petsc-users] DMPlex partition problem</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 7:25 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Wed, Apr 8, 2020 at 12:48 AM Danyang Su <<a href="mailto:danyang.su@gmail.com" target="_blank">danyang.su@gmail.com</a>> wrote:<o:p></o:p></p></div><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Dear All,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Hope you are safe and healthy.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>I have a question regarding pretty different partition results of prism mesh. The partition in PETSc generates much more ghost nodes/cells than the partition in Gmsh, even though both use metis as partitioner. Attached please find the prism mesh in both vtk and exo format, the test code modified based on ex1f90 example. Similar problem are observed for larger dataset with more layers.</span><o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I will figure this out by next week.<o:p></o:p></p></div></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have run your mesh and do not get those weird partitions. I am running in master. What are you using? Also, here is an easy way<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>to do this using a PETSc test:<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>cd $PETSC_DIR<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>make -f ./gmakefile test globsearch="dm_impls_plex_tests-ex1_cylinder" EXTRA_OPTIONS="-filename ${HOME}/Downloads/basin2layer.exo -dm_view hdf5:$PWD/mesh.h5 -dm_partition_view" NP=5 <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>./lib/petsc/bin/petsc_gen_xdmf.py mesh.h5<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>and then load mesh.xmf into Paraview. Here is what I see (attached). Is it possible for you to try the master branch?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Matt,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks for your quick response. If I use your script, the partition looks good, as shown in the attached figure. I am working on PETSc 3.13.0 release version on Mac OS. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Does the above script use code /petsc/src/dm/label/tutorials/ex1c.c?<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div></div></div></div></blockquote><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>It uses $PETSC_DIR/src/dm/impls/plex/tests/ex1.c<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I looked at your code and cannot see any difference. Also, no changes are in master that are not in 3.13. This is very strange.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I guess we will have to go one step at a time between the example and your code.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I will add mesh output to the ex1f90 example and check if the cell/vertex rank is exactly the same. I wrote the mesh output myself based on the partition but there should be no problem in that part. The number of ghost nodes and cells is pretty easy to check. Not sure if there is any difference between the C code and Fortran code that causes the problem. Anyway, I will keep you updated.<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Thanks,<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> Matt<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>For example, in Gmsh, I get partition results using two processors and four processors as shown below, which are pretty reasonable.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><img border=0 width=1243 height=632 style='width:12.9479in;height:6.5833in' id="gmail-m_-3435706974861814781gmail-m_-1037530915984171460gmail-m_-7031366058158567241gmail-m_1766518872680001549gmail-m_-6754249856326310125gmail-m_-5010994516531062944gmail-m_-3573090367149144785gmail-m_-3580117583950348177Picture_x0020_5" src="cid:image003.png@01D60DCD.996B4850"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><img border=0 width=1243 height=632 style='width:12.9479in;height:6.5833in' id="gmail-m_-3435706974861814781gmail-m_-1037530915984171460gmail-m_-7031366058158567241gmail-m_1766518872680001549gmail-m_-6754249856326310125gmail-m_-5010994516531062944gmail-m_-3573090367149144785gmail-m_-3580117583950348177Picture_x0020_6" src="cid:image004.png@01D60DCD.996B4850"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>However, in PETSc, the partition looks a bit weird. Looks like it takes layer partition first and then inside layer. If the number of nodes per layer is very large, this kind of partitioning results into much more ghost nodes/cells. </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Anybody know how to improve the partitioning in PETSc? I have tried parmetis and chaco. There is no big difference between them. </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US><img border=0 width=642 height=802 style='width:6.6875in;height:8.3541in' id="gmail-m_-3435706974861814781gmail-m_-1037530915984171460gmail-m_-7031366058158567241gmail-m_1766518872680001549gmail-m_-6754249856326310125gmail-m_-5010994516531062944gmail-m_-3573090367149144785gmail-m_-3580117583950348177Picture_x0020_8" src="cid:image005.png@01D60DCD.996B4850"></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Thanks,</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Danyang</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> </span><o:p></o:p></p></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><br clear=all><o:p></o:p></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></blockquote></div><p class=MsoNormal><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <o:p></o:p></p><div><div><div><div><div><div><div><p class=MsoNormal>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p></div></div></div></div></div></div></div></div></div></body></html>