<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Jed,<br>
<br>
<br>
On 5/1/2012 4:59 PM, Jed Brown wrote:
<blockquote
cite="mid:CAM9tzS=Oc91pydLz=WwrGQ8fj=yBhc_Mboms8AqYw1-7+1s0QQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">On Thu, Jan 5, 2012 at 09:41, TAY
wee-beng <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:zonexo@gmail.com">zonexo@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div id=":43s">I just did a -log_summary and attach the text
file, running across 8 and 16 processors. My most important
concern is whether the load is balanced across the
processors.<br>
<br>
In 16 processors case, for the time, it seems that the ratio
for many events are higher than 1, reaching up to 6.8 for
VecScatterEnd </div>
</blockquote>
<div><br>
</div>
<div>This takes about 1% of the run time and it's scaling well,
so don't worry about it.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div id=":43s">and 132.1 (?) for MatAssemblyBegin.</div>
</blockquote>
<div><br>
</div>
<div>This is about 2% of run time, but it's not scaling. Do you
compute a lot of matrix entries on processes that don't own
the rows?</div>
</div>
</blockquote>
<br>
I only compute rows which the processor own. Can it be the memory
allocation? I'll check on that. <br>
<blockquote
cite="mid:CAM9tzS=Oc91pydLz=WwrGQ8fj=yBhc_Mboms8AqYw1-7+1s0QQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div><br>
</div>
<div>Most of your solve time is going into PCSetUp() and
PCApply, both of which are getting more expensive as you add
processes. These are more than 10x more than spent in
MatMult() and MatMult() takes slightly less time on more
processes, so the increase isn't entirely due to memory
issues.</div>
<div><br>
</div>
<div>What methods are you using?</div>
</div>
</blockquote>
<br>
What do you mean methods? I am doing Cartesian grid 3d CFD, using
fractional mtd which solves the momentum and Poisson eqns. I
construct the linear eqn matrix and insert them in PETSc
matrix/vectors. Then I solve using Bicsstab and hypre AMG
respectively. Why is PCSetUp() and PCApply using more time?<br>
<blockquote
cite="mid:CAM9tzS=Oc91pydLz=WwrGQ8fj=yBhc_Mboms8AqYw1-7+1s0QQ@mail.gmail.com"
type="cite">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div id=":43s"> However, for the flops, ratios are 1 and 1.1.
so which is more important to look at? time or flops?</div>
</blockquote>
</div>
<br>
<div>If you would rather do a lot of flops than solve the problem
in a reasonable amount of time, you might as well use dense
methods. ;-)</div>
</blockquote>
<br>
Thanks again!<br>
</body>
</html>