<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Here is the bar chart I mentioned you should generate. As you can see the larger problem has 1.5096 extra seconds in VecTDot, 3.9548 extra seconds in Norm, is 1.266 seconds faster in Matmult, is 9.697 seconds slower in MatMultAdd, 1.047 secs slower
in MatMultTranspose and 5.006 seconds slower in MatSOR. All of these together match the total extra time of 19.64 well (19.94 compared to 19.64). </div>
<div class=""><br class="">
</div>
<div class=""> The Dot and Norm could be explained by using 8 times as many processes slowing down the reductions a bit. I cannot explain the slow down in MatSOR at all since that is embarrassingly parallel and should scale perfectly. I also cannot explain
the huge jump in MatMultAdd()!</div>
<div class=""><br class="">
</div>
<div class=""> This is why I ask you to run again to see if the numbers are consistent.</div>
<div class=""><br class="">
</div>
<div class=""> Barry</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<img apple-inline="yes" id="CB01EF12-45E0-4E3B-A305-98DEA1E7C0E7" width="640" height="319" src="cid:20ADD8C8-1DC4-4781-94E4-9C8AFC99F6C1@mcs.anl.gov" class=""><br class="">
<br class="">
<blockquote type="cite" class="">On May 29, 2018, at 6:18 AM, Michael Becker <<a href="mailto:Michael.Becker@physik.uni-giessen.de" class="">Michael.Becker@physik.uni-giessen.de</a>> wrote:<br class="">
<br class="">
Hello again,<br class="">
<br class="">
here are the updated log_view files for 125 and 1000 processors. I ran both problems twice, the first time with all processors per node allocated ("-1.txt"), the second with only half on twice the number of nodes ("-2.txt").<br class="">
<br class="">
<blockquote type="cite" cite="mid:079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov" class="">
<blockquote type="cite" class="">On May 24, 2018, at 12:24 AM, Michael Becker <<a href="mailto:Michael.Becker@physik.uni-giessen.de" class="">Michael.Becker@physik.uni-giessen.de</a>>
<div class=""> wrote:</div>
<div class=""><br class="">
</div>
<div class="">I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). </div>
<br class="Apple-interchange-newline">
</blockquote>
<div class=""> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and
destroy them in KSPReset() not during the solve. Not that this would be a measurable distance.</div>
<br class="Apple-interchange-newline">
</blockquote>
<br class="">
I mean this, right in the log_view output:<br class="">
<br class="">
<blockquote type="cite" class="">Memory usage is given in bytes:<br class="">
<br class="">
Object Type Creations Destructions Memory Descendants' Mem.<br class="">
Reports information only for process 0.<br class="">
<br class="">
--- Event Stage 0: Main Stage<br class="">
<br class="">
...<br class="">
<br class="">
--- Event Stage 1: First Solve<br class="">
<br class="">
...<br class="">
<br class="">
--- Event Stage 2: Remaining Solves<br class="">
<br class="">
Vector 23904 23904 1295501184 0.<br class="">
</blockquote>
I logged the exact number of KSP iterations over the 999 timesteps and its exactly 23904/6 = 3984.<br class="">
Michael<br class="">
<br class="">
<br class="">
Am 24.05.2018 um 19:50 schrieb Smith, Barry F.:<br class="">
<blockquote type="cite" cite="mid:079D8B4D-C57B-4054-B9BD-05644E52C3A4@anl.gov" class="">
<div class=""> Please send the log file for 1000 with cg as the solver.</div>
<div class=""><br class="">
</div>
<div class=""> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.)</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">On May 24, 2018, at 12:24 AM, Michael Becker <<a href="mailto:Michael.Becker@physik.uni-giessen.de" class="">Michael.Becker@physik.uni-giessen.de</a>>
<div class=""> wrote:</div>
<div class=""><br class="">
</div>
<div class="">I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES). </div>
<br class="Apple-interchange-newline">
</blockquote>
<div class=""> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and
destroy them in KSPReset() not during the solve. Not that this would be a measurable distance.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<div class="">This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here).</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance.</div>
<div class=""><br class="">
</div>
<div class="">Michael</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><log_view_125procs.txt><log_view_1000procs.txt></div>
<br class="Apple-interchange-newline">
</blockquote>
</blockquote>
<br class="">
<span id="cid:77B0BFFF-0CFE-46FC-9CCD-95B54A605A49@mcs.anl.gov"><log_view_125procs-1.txt></span><span id="cid:A26B1839-AD23-4E55-9802-9DA83D455BC8@mcs.anl.gov"><log_view_125procs-2.txt></span><span id="cid:3DC2ECFF-EA78-4ACC-AEF2-865298C287BD@mcs.anl.gov"><log_view_1000procs-1.txt></span><span id="cid:852ACC8D-B47E-403C-B9A6-297BB1E8FDF5@mcs.anl.gov"><log_view_1000procs-2.txt></span><br class="">
</blockquote>
<br class="">
</body>
</html>