<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Morten, <div><br></div><div>have a look at the local-to-global mapping that is created for the MATIS case in DMCreateMatrix_Plex in src/dm/impls/plex/plex.c.</div><div>and how DMPlexMatSetClosure is used in src/snes/utils/dmplexsnes.c in the MATIS case.</div><div><div><br></div><div>MATIS is a format for non-overlapping domain decomposition when you assemble your stiffness matrix on each subdomain (MPI proc) separately.</div><div><br></div><div><div>On Sep 26, 2016, at 5:04 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Sep 26, 2016 at 8:41 AM, Morten Nobel-Jørgensen<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;"><div>Hi Matt</div><div><br></div><div><div>We are trying to do a simple FE using DMPlex, but when assemble the global stiffness matrix we get in problems when running NP>1 - that is the global matrix differs when we move to a distributed system, where it should not.</div><div><br></div><div>In pseudo-code our CreateGlobalStiffnessMatrix does the following</div></div><blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;"><div><font face="Courier New">create a local stiffness matrix ke with some values</font></div><div><font face="Courier New">for each local cell/element e (using result from DMPlexGetHeightStratum(..,0,..<wbr>)</font></div><div><font face="Courier New"> <span class="Apple-converted-space"> </span>for each of its vertices (using DMPlexGetTransitiveClosure(..,<wbr>e,..)</font></div><div><font face="Courier New"> <span class="Apple-converted-space"> </span>set local/global mapping to edof</font></div><div><font face="Courier New">update the global stiffness matrix K using the local mapping edof and values ke</font></div></blockquote><div><div><br></div><div>The code we have sent is a simplified version, which just builds a dummy stiffness matrix - but we believe this matrix should still the same independent of NP. (That is why we use trace).</div></div></div></blockquote><div><br></div><div>I am not sure what is wrong there, but there are a bunch of Plex FEM examples, like SNES ex12, ex62, ex77.</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;">I'm not familiar with MatSetValuesClosure(). Is that the missing piece?</div></blockquote><div><br></div><div>I use this to do indexing since it is so error prone to do it by yourself. I think this could be the problem with your example.</div><div><br></div><div>When you distribute the mesh, ordering changes, and we do permutations to keep all local indices contiguous. The</div><div>MatSetValuesClosure() is a convenience function for FEM which takes in a mesh point (could be cell, face, etc.) and</div><div>translates that point number to a set of row indices using</div><div><br></div><div> a) the transitive closure of that point in the mesh DAG</div><div><br></div><div>and</div><div><br></div><div> b) the default PetscSection for the DM which maps mesh points to sets of dofs (row indices)</div><div><br></div><div>For it to work, you need to setup the default DM section, which it looks like you have done.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;"><div><div><span style="font-size: 10pt;">Kind regards,</span></div><div>Morten</div></div><div><br></div><br><div style="font-family: 'Times New Roman'; font-size: 16px;"><hr><div style="direction: ltr;"><font face="Tahoma" size="2"><b>From:</b><span class="Apple-converted-space"> </span>Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br><b>Sent:</b><span class="Apple-converted-space"> </span>Monday, September 26, 2016 2:19 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Morten Nobel-Jørgensen<br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc [<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>]<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] DMPlex problem<br></font><br></div><div></div><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Sep 26, 2016 at 7:00 AM, Morten Nobel-Jørgensen<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;">Hi Matthew<br><br>It seems like the problem is not fully fixed. I have changed the code to now run on with both 2,3 and 4 cells. When I run the code using NP = 1..3 I get different result both for NP=1 to NP=2/3 when cell count is larger than 2.<br></div></div></blockquote><div><br></div><div>Do you mean the trace? I have no idea what you are actually putting in.</div><div><br></div><div>I have a lot of debugging when you use, MatSetValuesClosure(), but when you directly use</div><div>MatSetValuesLocal(), you are handling things yourself.</div><div><br></div><div> <span class="Apple-converted-space"> </span>Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;"><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;">Kind regards,<br>Morten<br>____<br>mpiexec -np 1 ./ex18k<br>cells 2<br>Loc size: 36<br>Trace of matrix: 132.000000<br>cells 3<br>Loc size: 48<br>Trace of matrix: 192.000000<br>cells 4<br>Loc size: 60<br>Trace of matrix: 258.000000<br>mpiexec -np 2 ./ex18k<br>cells 2<br>Loc size: 24<br>Loc size: 24<br>Trace of matrix: 132.000000<br>cells 3<br>Loc size: 36<br>Loc size: 24<br>Trace of matrix: 198.000000<br>cells 4<br>Loc size: 36<br>Loc size: 36<br>Trace of matrix: 264.000000<br>mpiexec -np 3 ./ex18k<br>cells 2<br>Loc size: 24<br>Loc size: 24<br>Loc size: 0<br>Trace of matrix: 132.000000<br>cells 3<br>Loc size: 24<br>Loc size: 24<br>Loc size: 24<br>Trace of matrix: 198.000000<br>cells 4<br>Loc size: 36<br>Loc size: 24<br>Loc size: 24<br>Trace of matrix: 264.000000<br><br><br><br><br><div style="font-family: 'Times New Roman'; font-size: 16px;"><hr><div style="direction: ltr;"><font size="2" face="Tahoma"><b>From:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.go<wbr>v</a><span class="Apple-converted-space"> </span>[<a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.g<wbr>ov</a>] on behalf of Morten Nobel-Jørgensen [<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>]<br><b>Sent:</b><span class="Apple-converted-space"> </span>Sunday, September 25, 2016 11:15 AM<br><b>To:</b><span class="Apple-converted-space"> </span>Matthew Knepley<br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc [<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>]<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] DMPlex problem<br></font><br></div><div></div><div><div style="direction: ltr; font-family: Tahoma; font-size: 10pt;">Hi Matthew<div><br></div><div>Thank you for the bug-fix :) I can confirm that it works :)</div><div><br></div><div>And thanks for your hard work on PETSc - your work is very much appreciated!</div><div><br></div><div>Kind regards,</div><div>Morten<br><div style="font-family: 'Times New Roman'; font-size: 16px;"><hr><div style="direction: ltr;"><font size="2" face="Tahoma"><b>From:</b><span class="Apple-converted-space"> </span>Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, September 23, 2016 2:46 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Morten Nobel-Jørgensen<br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc [<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>]<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] DMPlex problem<br></font><br></div><div></div><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Sep 23, 2016 at 7:45 AM, Matthew Knepley<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Fri, Sep 23, 2016 at 3:48 AM, Morten Nobel-Jørgensen<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>></span><span class="Apple-converted-space"> </span>wrote:<br></span><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="direction: ltr; font-family: tahoma; font-size: 10pt;"><div><span style="font-size: 10pt;">Dear PETSc developers</span></div><div><br></div><span>Any update on this issue regarding DMPlex? Or is there any obvious workaround that we are unaware of?</span></div></blockquote><div><br></div><div>I have fixed this bug. It did not come up in nightly tests because we are not using MatSetValuesLocal(). Instead we</div><div>use MatSetValuesClosure() which translates differently.</div><div><br></div><div>Here is the branch</div><div><br></div><div> <a href="https://bitbucket.org/petsc/petsc/branch/knepley/fix-dm-ltog-bs" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/branch/knepley/fix-dm-lt<wbr>og-bs</a></div><div><br></div><div>and I have merged it to next. It will go to master in a day or two.</div></div></div></div></blockquote><div><br></div><div>Also, here is the cleaned up source with no memory leaks.</div><div><br></div><div> <span class="Apple-converted-space"> </span>Matt</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="direction: ltr; font-family: tahoma; font-size: 10pt;">Also should we additionally register the issue on Bitbucket or is reporting the issue on the mailing list enough?</div></blockquote><div><br></div></span><div>Normally we are faster, but the start of the semester was hard this year.</div><div><br></div><div> <span class="Apple-converted-space"> </span>Thanks,</div><div><br></div><div> Matt</div><div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="direction: ltr; font-family: tahoma; font-size: 10pt;"><div>Kind regards,</div><div>Morten</div><br><div style="font-family: 'times new roman'; font-size: 16px;"><span><hr><div style="direction: ltr;"><font size="2" face="Tahoma"><b>From:</b><span class="Apple-converted-space"> </span>Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, September 09, 2016 12:21 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Morten Nobel-Jørgensen<br><b>Cc:</b><span class="Apple-converted-space"> </span>PETSc [<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>]<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] DMPlex problem<br></font><br></div><div></div></span><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Sep 9, 2016 at 4:04 AM, Morten Nobel-Jørgensen<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="direction: ltr; font-family: tahoma; font-size: 10pt;">Dear PETSc developers and users,<br><br>Last week we posted a question regarding an error with DMPlex and multiple dofs and have not gotten any feedback yet. This is uncharted waters for us, since we have gotten used to an extremely fast feedback from the PETSc crew. So - with the chance of sounding impatient and ungrateful - we would like to hear if anybody has any ideas that could point us in the right direction?<br></div></blockquote><div><br></div><div>This is my fault. You have not gotten a response because everyone else was waiting for me, and I have been</div><div>slow because I just moved houses at the same time as term started here. Sorry about that.</div><div><br></div><div>The example ran for me and I saw your problem. The local-tp-global map is missing for some reason.</div><div>I am tracking it down now. It should be made by DMCreateMatrix(), so this is mysterious. I hope to have</div><div>this fixed by early next week.</div><div><br></div><div> <span class="Apple-converted-space"> </span>Thanks,</div><div><br></div><div> <span class="Apple-converted-space"> </span>Matt</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="direction: ltr; font-family: tahoma; font-size: 10pt;">We have created a small example problem that demonstrates the error in the matrix assembly.<br><div><font size="1"><span style="font-size: 13px;"><font size="2"><br>Thanks,<br>Morten</font></span></font></div><div><font size="1"><span style="font-size: 13px;"><font size="2"><br></font></span></font></div><div><font size="1"><span style="font-size: 13px;"><font size="2"><br></font></span></font></div></div></blockquote></div><br><br clear="all"><div><br></div>--<span class="Apple-converted-space"> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></div></div></div></div></div></blockquote></div></div><div><div><br><br clear="all"><span><font color="#888888"><div><br></div>--<span class="Apple-converted-space"> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></div><span><font color="#888888"></font></span></div><span><font color="#888888"></font></span></div><span><font color="#888888"></font></span></div><span><font color="#888888"></font></span></blockquote><span><font color="#888888"></font></span></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>--<span class="Apple-converted-space"> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"></font></span></blockquote><span class="HOEnZb"><font color="#888888"></font></span></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>--<span class="Apple-converted-space"> </span><br><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></font></span></div></div></div></div></div></blockquote></div><br><br clear="all"><div><br></div>--<span class="Apple-converted-space"> </span><br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div></div></div></div></blockquote></div><br></div></body></html>