<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">petsc-users <petsc-users-bounces@mcs.anl.gov> on behalf of Matthew Knepley via petsc-users <petsc-users@mcs.anl.gov><br>
<b>Reply-To: </b>Matthew Knepley <knepley@gmail.com><br>
<b>Date: </b>Wednesday, May 8, 2019 at 7:46 AM<br>
<b>To: </b>Justin Chang <jychang48@gmail.com><br>
<b>Cc: </b>petsc-users <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] Extremely slow DMNetwork Jacobian assembly<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">On Wed, May 8, 2019 at 4:45 AM Justin Chang via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi guys, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I have a fully working distribution system solver written using DMNetwork, The idea is that each electrical bus can have up to three phase nodes, and each phase node has two unknowns: voltage magnitude and angle.
In a completely balanced system, each bus has three nodes, but in an unbalanced system some of the buses can be either single phase or two-phase.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">The working DMNetwork code I developed, loosely based on the SNES network/power.c, essentially represents each vertex as a bus. DMNetworkAddNumVariables() function will add either 2, 4, or 6 unknowns to each vertex.
If every single bus had the same number of variables, the mat block size = 2, 4, or 6, and my code is both fast and scalable. However, if the unknowns per DMNetwork vertex unknowns are not the same across, then my SNESFormJacobian function becomes extremely
extremely slow. Specifically, the MatSetValues() calls when the col/row global indices contain an offset value that points to a neighboring bus vertex. <o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I have never seen MatSetValues() be slow unless it is allocating. Did you confirm that you are not allocating, with -info?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> Matt<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have written power grid codes using DMNetwork where the vertex dofs range from 2 to 20. I have not yet observed the slow-down you report. My guess, as Matt points, is something to do with the preallocation.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In power.c example, the DM creates the Jacobian matrix (which sets the Jacobian nonzero structure and does the allocation). Do you have the following lines in your code?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:#FFFFFE"><span style="font-size:10.0pt;font-family:Menlo;color:#202020">ierr</span><span style="font-size:10.0pt;font-family:Menlo;color:black"> =
</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">DMCreateMatrix</span><span style="font-size:10.0pt;font-family:Menlo;color:black">(</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">networkdm</span><span style="font-size:10.0pt;font-family:Menlo;color:black">,&</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">J</span><span style="font-size:10.0pt;font-family:Menlo;color:black">);</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">CHKERRQ</span><span style="font-size:10.0pt;font-family:Menlo;color:black">(</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">ierr</span><span style="font-size:10.0pt;font-family:Menlo;color:black">);<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt;background:#FFFFFE"><span style="font-size:10.0pt;font-family:Menlo;color:#202020">ierr</span><span style="font-size:10.0pt;font-family:Menlo;color:black"> =
</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">MatSetOption</span><span style="font-size:10.0pt;font-family:Menlo;color:black">(</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">J</span><span style="font-size:10.0pt;font-family:Menlo;color:black">,</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">MAT_NEW_NONZERO_ALLOCATION_ERR</span><span style="font-size:10.0pt;font-family:Menlo;color:black">,</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">PETSC_FALSE</span><span style="font-size:10.0pt;font-family:Menlo;color:black">);</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">CHKERRQ</span><span style="font-size:10.0pt;font-family:Menlo;color:black">(</span><span style="font-size:10.0pt;font-family:Menlo;color:#202020">ierr</span><span style="font-size:10.0pt;font-family:Menlo;color:black">);<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Why is that? Is it because I no longer have a uniform block structure and lose the speed/optimization benefits of iterating through an AIJ matrix? I see three potential workarounds:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">1) Treat every vertex as a three phase bus and "zero out" all the unused phase node dofs and put a 1 in the diagonal. The problem I see with this is that I will have unnecessary degrees of freedom (aka non-zeros
in the matrix). From the distribution systems I've seen, it's possible that anywhere from 1/3 to 1/2 of the buses will be two-phase or less, meaning I may have nearly twice the amount of dofs than necessary if I wanted to preserve the block size = 6 for the
AU mat.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">2) Treat every phase node as a vertex aka solve a single-phase power flow solver. That way I guarantee to have a block size = 2, this is what Domenico's former student did in his thesis work. The problem I see with
this is that I have a larger graph, which can take more time to setup and parallelize.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">3) Create a "fieldsplit" where I essentially have three "blocks" - one for buses with all three phases, another for buses with only two phases, one for single-phase buses. This way each block/fieldsplit will have
a consistent block size. I am not sure if this will solve the MatSetValues() issues, but it's, but can anyone give pointers on how to go about achieving this?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Justin<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>