<div class="gmail_quote">On Mon, Jan 9, 2012 at 09:57, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com">zonexo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">I only compute rows which the processor own. Can it be the memory
allocation? I'll check on that. <br></div></blockquote><div><br></div><div>Usually memory allocation mistakes cost much more.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div class="im">
<blockquote type="cite">
<div class="gmail_quote">
<div><br>
</div>
<div>Most of your solve time is going into PCSetUp() and
PCApply, both of which are getting more expensive as you add
processes. These are more than 10x more than spent in
MatMult() and MatMult() takes slightly less time on more
processes, so the increase isn't entirely due to memory
issues.</div>
<div><br>
</div>
<div>What methods are you using?</div>
</div>
</blockquote>
<br></div>
What do you mean methods? I am doing Cartesian grid 3d CFD,</div></blockquote><div><br></div><div>Then just use a regular partition. If your meshes have boundary layers, you might want to adjust the subdomain aspect ratios so that strongly coupled cells tend to reside on the same process, but don't bother with general graph partitions (like ParMETIS) because you will end up writing most of an unstructured CFD code in order to use those partitions efficiently.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"> using
fractional mtd which solves the momentum and Poisson eqns. I
construct the linear eqn matrix and insert them in PETSc
matrix/vectors. Then I solve using Bicsstab and hypre AMG
respectively. Why is PCSetUp() and PCApply using more time?</div></blockquote></div><br><div>It is expensive because BoomerAMG setup and apply is expensive.</div>