<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi Sir,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
MatScale in "<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Main Stage" is indeed called 6 times for 0% run time. In stage "Stiff_Adj" though, we get :</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">MatScale 8192 1.0 7.1185e+01 1.0 3.43e+10 1.0 0.0e+00 0.0e+00 0.0e+00 50 46 0 0 0 80 98 0 0 0 482<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">MatMult is indeed expensive (23% run time) and should be improved, but <span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">MatScale<span> in <span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;">"Stiff_Adj"<span> is
still taking 50% run time</span></span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><br>
</span></span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span>Thanks,</span></span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><br>
</span></span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span><span style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); display: inline !important;"><span>Antoine</span></span></span></span></span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> Barry Smith <bsmith@petsc.dev><br>
<b>Envoyé :</b> 22 octobre 2020 16:09<br>
<b>À :</b> Antoine Côté <Antoine.Cote3@USherbrooke.ca><br>
<b>Cc :</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Objet :</b> Re: [petsc-users] Enhancing MatScale computing time</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><br class="">
</div>
<div class="">MatMult 9553 1.0 3.2824e+01 1.0 3.54e+10 1.0 0.0e+00 0.0e+00 0.0e+00 23 48 0 0 0 61 91 0 0 0 1079</div>
<div class="">MatScale 6 1.0 5.3896e-02 1.0 2.52e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 467</div>
<div class=""><br class="">
</div>
<div>Though the flop rate of MatScale is not so high (467) it is taking very little (0 percent of the run time while MatMult takes 23 percent of the time).</div>
<div><br class="">
</div>
<div>So the main cost related to the matrices is MatMult because it has a lot of operations 9553, you might think about your algorithms you are using and if there </div>
<div>improvements. </div>
<div><br class="">
</div>
<div>It looks like you are using some kind of multigrid and solve 6 problems with 1357 total iterations which is 200 iterations per solve. This is absolutely HUGE for multigrain, you need to tune the multigrid for you problem to bring that down to at most a
couple dozen iterations per solve.</div>
<div><br class="">
</div>
<div> Barry</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 22, 2020, at 3:02 PM, Antoine Côté <<a href="mailto:Antoine.Cote3@USherbrooke.ca" class="">Antoine.Cote3@USherbrooke.ca</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
Hi,</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
See attached files for both outputs. Tell me if you need any clarification. It was run with a DMDA of 33x17x17 nodes (creating 32x16x16=8192 elements). With 3 dof per nodes, problem has a total of 28611 dof.</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
Note : Stage "Stiff_Adj" is the part of the code modifying Mat K. PetscLogStagePush/Pop was used.</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
Regards,</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
Antoine</div>
<div id="x_appendonsend" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; display:inline-block; width:1138.75px">
<span class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" class="" style="font-size:11pt"><b class="">De :</b><span class="x_Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>><br class="">
<b class="">Envoyé :</b><span class="x_Apple-converted-space"> </span>22 octobre 2020 15:35<br class="">
<b class="">À :</b><span class="x_Apple-converted-space"> </span>Antoine Côté <<a href="mailto:Antoine.Cote3@USherbrooke.ca" class="">Antoine.Cote3@USherbrooke.ca</a>><br class="">
<b class="">Cc :</b><span class="x_Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a><span class="x_Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Objet :</b><span class="x_Apple-converted-space"> </span>Re: [petsc-users] Enhancing MatScale computing time</font>
<div class=""> </div>
</div>
<div class="" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="ltr" class="">
<div dir="ltr" class="">On Thu, Oct 22, 2020 at 3:23 PM Antoine Côté <<a href="mailto:Antoine.Cote3@usherbrooke.ca" class="">Antoine.Cote3@usherbrooke.ca</a>> wrote:<br class="">
</div>
<div class="x_x_gmail_quote">
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
Hi,</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
I'm working with a 3D DMDA, with 3 dof per "node", used to create a sparse matrix Mat K. The Mat is modified repeatedly by the program, using the commands (in that order) :</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
MatZeroEntries(K)<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
In a for loop : MatSetValuesLocal(K, 24, irow, 24, icol, vals, ADD_VALUES)</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
MatAssemblyBegin(K, MAT_FINAL_ASSEMBLY)<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<span class="" style="background-color:rgb(255,255,255); display:inline">MatAssemblyEnd(K, MAT_FINAL_ASSEMBLY)</span><br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
MatDiagonalScale(K, vec1, vec1)<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
MatDiagonalSet(K, vec2, ADD_VALUES)<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<span class="" style="background-color:rgb(255,255,255); display:inline">Computing time seems high and I</span><span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"> would like to improve it. Running tests with "-log_view" tells
me that MatScale() is the bottle neck (50% of total computing time) . From manual pages, I've tried a few tweaks :</span></div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<ul class="">
<li class="">DMSetMatType(da, MATMPIBAIJ) : "For problems with multiple degrees of freedom per node, ... BAIJ can significantly enhance performance", Chapter 14.2.4<br class="">
</li><li class="">Used MatMissingDiagonal() to confirm there is no missing diagonal entries : "If the matrix Y is missing some diagonal entries this routine can be very slow", MatDiagonalSet() manual</li><li class="">Tried MatSetOption()</li><ul class="">
<li class="">MAT_NEW_NONZERO_LOCATIONS == PETSC_FALSE : to increase assembly efficiency</li><li class="">MAT_NEW_NONZERO_LOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br class="">
</li><li class="">MAT_NEW_NONZERO_ALLOCATION_ERR == PETSC_TRUE : "When true, assembly processes have one less global reduction"<br class="">
</li><li class="">MAT_USE_HASH_TABLE == PETSC_TRUE : "Improve the searches during matrix assembly"<br class="">
</li></ul>
</ul>
<div class=""><span class="">According to<span class="x_Apple-converted-space"> </span></span><span class="" style="background-color:rgb(255,255,255); display:inline">"-log_view"</span><span class="">, assembly is fast (0% of total time), and the use of a DMDA
makes me believe preallocation isn't the cause of performance issue.</span><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">I would like to know how could I improve <span class="" style="background-color:rgb(255,255,255); display:inline">MatScale(). W</span>hat are the best practices
(during allocation, when defining Vecs and Mats, the DMDA, etc.)? Instead of <span class="" style="background-color:rgb(255,255,255); display:inline">MatDiagonalScale(), s<span class="" style="background-color:rgb(255,255,255); display:inline">hould I use
another command<span class=""> </span></span>to obtain the same result faster?</span></span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Something is definitely strange. Can you please send the output of</div>
<div class=""><br class="">
</div>
<div class=""> -log_view -info :mat</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<div class=""><span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Thank you very much!</span></div>
<div class=""><span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br class="">
</span></div>
<div class=""><span class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Antoine Côté</span></div>
</div>
<div class="" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; background-color:rgb(255,255,255)">
<br class="">
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="x_Apple-converted-space"> </span><br class="">
<div dir="ltr" class="x_x_gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://can01.safelinks.protection.outlook.com/?url=http:%2F%2Fwww.cse.buffalo.edu%2F~knepley%2F&data=04%7C01%7CAntoine.Cote3%40usherbrooke.ca%7C2f4d6ff4e9aa48b4058a08d876c6665d%7C3a5a8744593545f99423b32c3a5de082%7C0%7C0%7C637389941843624498%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EgZu%2BdmuXzZwE8LSyMC4BhoC7Or%2BHvrwykv%2BcPZOCXg%3D&reserved=0" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="JMllj9gkibn6/an7pmVZ3HaSdPVpX23gyFQRDLxXkL0tMf3lD86JWCGa9jWH0C4IibawgdXaRVGBAtr5D4Vt4gAa5jmE7TOZALZLWVZ0HDosZObxkfNSTcw7J3o9mS0FDOtJP436QRMMQvFzdAxCF2u3FD7O6djuQdK9qs80QZI=" originalsrc="http://www.cse.buffalo.edu/~knepley/" shash="A5TjyYp384mqAvon0t9PnnYoeIvNxn7rXoeq1ZpGAU5jQLwdr/TkqUejSYs09W94qjJB2V6IlQISWWPmO+ow7uDPEC+T1NcrsZ/dKBWpj+KFYJGOu+LfZ6kIN24RcMZVWz1owUXpQfAi9vy4eyCZt+xAJpFEDU3vGn+LfVFi3h8=" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span id="x_cid:92E7F4D1-2260-4DCD-9A47-B11ECD816830@hsd1.il.comcast.net"><LogView.out></span><span id="x_cid:18BE52CE-82ED-446C-9774-1E9A8F4B26AF@hsd1.il.comcast.net"><mat.0></span></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>