<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body style="word-wrap:break-word; color:rgb(0,0,0); font-size:14px; font-family:Calibri,sans-serif" fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">
<div>We have now hit a related problem. If we change the dof from 1 to 3 we the following error message. I'm using the next branch (pulled from git today).</div>
<div><br>
</div>
<div>
<div>mpiexec -np 2 ./ex18k</div>
<div>[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div>
<div>[1]PETSC ERROR: Argument out of range</div>
<div>[1]PETSC ERROR: Row too large: row 36 max 35</div>
<div>[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.</div>
<div>[1]PETSC ERROR: Petsc Development GIT revision: v3.7.3-1857-g5b40c63 GIT Date: 2016-08-29 22:13:25 -0500</div>
<div>[1]PETSC ERROR: ./ex18k on a arch-linux2-c-debug named morten-VirtualBox by morten Tue Aug 30 21:31:36 2016</div>
<div>[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --download-netcdf --download-hdf5 --download-exodusii</div>
<div>[1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 556 in /home/morten/petsc/src/mat/impls/aij/mpi/mpiaij.c</div>
<div>[1]PETSC ERROR: #2 MatSetValues() line 1239 in /home/morten/petsc/src/mat/interface/matrix.c</div>
<div>[1]PETSC ERROR: #3 MatSetValuesLocal() line 2102 in /home/morten/petsc/src/mat/interface/matrix.c</div>
<div>[1]PETSC ERROR: #4 CreateGlobalStiffnessMatrix() line 82 in /home/morten/topop_in_petsc_unstruct/ex18k.cc</div>
<div>[1]PETSC ERROR: #5 main() line 113 in /home/morten/topop_in_petsc_unstruct/ex18k.cc</div>
<div>[1]PETSC ERROR: No PETSc Option Table entries</div>
<div><br>
</div>
<div>We are not fully sure if this is related to the way we use DMPlex or a bug in our code. Any help or tips will be appreciated :)</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Morten</div>
<div><br>
</div>
<div><br>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF376638" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Morten Nobel-Jørgensen<br>
<b>Sent:</b> Thursday, July 14, 2016 9:45 AM<br>
<b>To:</b> Matthew Knepley<br>
<b>Cc:</b> petsc-users@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-users] Distribution of DMPlex for FEM<br>
</font><br>
</div>
<div></div>
<div>
<div>Hi Matthew</div>
<div><br>
</div>
<div>Thanks for your answer and your fix. It works :)))</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Morten</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; border-bottom:medium none; border-left:medium none; padding-bottom:0in; padding-left:0in; padding-right:0in; border-top:#b5c4df 1pt solid; border-right:medium none; padding-top:3pt">
<span style="font-weight:bold">Fra: </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<span style="font-weight:bold">Dato: </span>Thursday 14 July 2016 at 00:03<br>
<span style="font-weight:bold">Til: </span>Morten Nobel-Joergensen <<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Emne: </span>Re: [petsc-users] Distribution of DMPlex for FEM<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Jul 13, 2016 at 3:57 AM, Morten Nobel-Jørgensen <span dir="ltr">
<<a href="mailto:mono@dtu.dk" target="_blank">mono@dtu.dk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">I’m having problems distributing a simple FEM model using DMPlex. For test case I use 1x1x2 hex box elements (/cells) with 12 vertices. Each vertex has one DOF.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">When I distribute the system to two processors, each get a single element and the local vector has the size 8 (one DOF for each vertex of a hex box) as expected.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">My problem is that when I manually assemble the global stiffness matrix (a 12x12 matrix) it seems like my ghost values are ignored. I’m sure that I’m missing something obvious but
cannot see what it is.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">In the attached example, I’m assembling the global stiffness matrix using a simple local stiffness matrix of ones. This makes it very easy to see if the matrix is assembled correctly.
If I run it on one process, then global stiffness matrix consists of 0’s, 1’s and 2’s and its trace is 16.0. But if I run it distributed on on two, then it consists only of 0's and 1’s and its trace is 12.0.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px"><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">I hope that somebody can spot my mistake and help me in the right direction :)</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is my fault, and Stefano Zampini had already tried to tell me this was broken. I normally use DMPlexMatSetClosure(), which handles global indices correctly.</div>
<div>I have fixed this in the branch</div>
<div><br>
</div>
<div> knepley/fix-plex-l2g</div>
<div><br>
</div>
<div>which is also merged to 'next'. I am attaching a version of your sample where all objects are freed correctly. Let me know if that works for you.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Kind regards,</div>
<div style="color:rgb(0,0,0); font-family:Calibri,sans-serif; font-size:14px">Morten</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</body>
</html>