<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
See attached files for both outputs. Tell me if you need any clarification. It was run with a DMDA of 33x17x17 nodes (creating 32x16x16=8192 elements). With 3 dof per nodes, problem has a total of 28611 dof.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Note : Stage "Stiff_Adj" is the part of the code modifying Mat K. PetscLogStagePush/Pop was used.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Antoine</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> Matthew Knepley <knepley@gmail.com><br>
<b>Envoyé :</b> 22 octobre 2020 15:35<br>
<b>À :</b> Antoine Côté <Antoine.Cote3@USherbrooke.ca><br>
<b>Cc :</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Objet :</b> Re: [petsc-users] Enhancing MatScale computing time</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Thu, Oct 22, 2020 at 3:23 PM Antoine Côté <<a href="mailto:Antoine.Cote3@usherbrooke.ca">Antoine.Cote3@usherbrooke.ca</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
I'm working with a 3D DMDA, with 3 dof per "node", used to create a sparse matrix Mat K. The Mat is modified repeatedly by the program, using the commands (in that order) :</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
MatZeroEntries(K)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
In a for loop : MatSetValuesLocal(K, 24, irow, 24, icol, vals, ADD_VALUES)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
MatAssemblyBegin(K, MAT_FINAL_ASSEMBLY)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">MatAssemblyEnd(K, MAT_FINAL_ASSEMBLY)</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
MatDiagonalScale(K, vec1, vec1)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
MatDiagonalSet(K, vec2, ADD_VALUES)<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">Computing time seems high and I</span><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"> would like to improve it. Running tests with "-log_view" tells
me that MatScale() is the bottle neck (50% of total computing time) . From manual pages, I've tried a few tweaks :</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<ul>
<li>DMSetMatType(da, MATMPIBAIJ) : "For problems with multiple degrees of freedom per node, ... BAIJ can significantly enhance performance", Chapter 14.2.4<br>
</li><li>Used MatMissingDiagonal() to confirm there is no missing diagonal entries : "If the matrix Y is missing some diagonal entries this routine can be very slow", MatDiagonalSet() manual</li><li>Tried MatSetOption()</li><ul>
<li>MAT_NEW_NONZERO_LOCATIONS == PETSC_FALSE : to increase assembly efficiency</li><li>MAT_NEW_NONZERO_LOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br>
</li><li>MAT_NEW_NONZERO_ALLOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br>
</li><li>MAT_USE_HASH_TABLE == PETSC_TRUE : "Improve the searches during matrix assembly"<br>
</li></ul>
</ul>
<div><span>According to </span><span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">"-log_view"</span><span>, assembly is fast (0% of total time), and the use of a DMDA makes me believe preallocation isn't the cause of performance
issue.</span><br>
</div>
<div><br>
</div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">I would like to know how could I improve <span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">MatScale(). W</span>hat are the best practices (during
allocation, when defining Vecs and Mats, the DMDA, etc.)? Instead of <span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">MatDiagonalScale(), s<span style="color:rgb(0,0,0); background-color:rgb(255,255,255); display:inline">hould
I use another command<span> </span></span>to obtain the same result faster?</span></span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Something is definitely strange. Can you please send the output of</div>
<div><br>
</div>
<div> -log_view -info :mat</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Thank you very much!</span></div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Antoine Côté</span></div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://can01.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=04%7C01%7CAntoine.Cote3%40usherbrooke.ca%7C6b823852b3964170f52908d876c1bb0b%7C3a5a8744593545f99423b32c3a5de082%7C0%7C0%7C637389921724846720%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=w7%2Fc%2BSzAfTa02gxTS8VbB%2FVwIPpaKw%2F%2BeiiX4K9gd1k%3D&reserved=0" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="A5TjyYp384mqAvon0t9PnnYoeIvNxn7rXoeq1ZpGAU5jQLwdr/TkqUejSYs09W94qjJB2V6IlQISWWPmO+ow7uDPEC+T1NcrsZ/dKBWpj+KFYJGOu+LfZ6kIN24RcMZVWz1owUXpQfAi9vy4eyCZt+xAJpFEDU3vGn+LfVFi3h8=" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>