<div dir="ltr">AMG setup is kind of expensive, say the order of a solve. What you have looks OK here. If you have a super hard problem you will want to coarsen slower (high threshold), which will increase setup costs. The setup costs are symbolic (graph work) and numeric (like a factorization). As you noticed:<div><br></div><div><pre class="">MatPtAPNumeric 4 1.0 1.3035e-01 1.0 4.64e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 356
</pre></div><div><br></div><div><br></div><div>This is the numeric part. The symbolic part will get amortized if the grid does not change and the numerical part will get amortized if the operator does not change (linear).</div><div><br></div><div>Mark</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 4, 2015 at 5:47 PM, Justin Chang <span dir="ltr"><<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thank you Matt and Mark for the clarification. Matt, if you recall our discussion about calculating the arithmetic intensity from the earlier threads, it seems GAMG now has a myriad of all these additional vector and matrix operations that were not present in the CG/Jacobi case. Running with the command line options you and Mark suggested, I now have these additional operations to deal with:<br><br></div><div>VecMDot<br></div><div>VecAXPBYCZ<br></div><div>VecMAXPY<br></div><div>VecSetRandom<br></div><div>VecNormalize<br></div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div>MatMultAdd<br></div>MatMultTranspose<br></div>MatSolve<br></div>MatConvert<br></div>MatScale<br></div>MatResidual<br></div><div>MatCoarsen<br></div><div>MatAXPY<br></div>MatMatMult<br></div>MatMatMultSym<br></div>MatMatMultNum<br></div>MatPtAP<br></div>MatPtAPSymbolic<br></div>PatPtAPNumeric<br></div>MatTrnMatMult<br></div>MatTrnMatMultSym<br></div>MatTrnMatMultNum<br></div>MatGetSymTrans<br></div>KSPGMRESOrthog<br></div>PCGAMGGraph_AGG<br></div>PCGAMGCoarse_AGG<br></div>PCGAMGProl_AGG<br></div>PCGAMGPOpt_AGG<br></div>GAMG: createProl and all of its associated events.<br></div><div>GAMG: partLevel</div><div>PCSetUpOnBlocks<br><br></div><div>Attached is the output from -log_summary showing the exact counts for the case I am running.<br><br></div><div>I have the following questions:<br><br></div><div>1) For the Vec operations VecMDot and VecMAXPY, it seems the estimation of total bytes transferred (TBT) relies on knowing how many vectors there are. Is there a way to figure this out? Or at least with gamg what would it be, three vectors?<br><br></div><div>2) It seems there are a lot of matrix manipulations and multiplications. Is it safe to say that the size and number of non zeroes is the same? Or will it change?<br><br></div><div>3) If I follow the TBT tabulation as in that paper you pointed me to, would MatMultTranspose follow the same formula if the Jacobian is symmetric?<br><br></div><div>4) How do I calculate anything that requires the multiplication of at least two matrices?<br><br></div><div>5) More importantly, are any of the above calculations necessary? Because log_summary seems to indicate that MatMult() has the greatest amount of workload and number of calls. My only hesitation is how much traffic MatMatMults may take (assuming I go off of the same assumptions as in that paper).<br><br></div><div>6) And/or, are there any other functions that I missed that might be important to calculate as well?<br></div><div><br></div><div>Thanks,<br></div><div>Justin<br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 4, 2015 at 11:33 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 4, 2015 at 12:29 PM, Matthew Knepley <span dir="ltr"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Thu, Jun 4, 2015 at 10:31 AM, Justin Chang <span dir="ltr"><<a href="mailto:jychang48@gmail.com" target="_blank">jychang48@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Yeah I saw his recommendation and am trying it out. But I am not sure what most of those parameters mean. For instance:<br><br></div>1) What does -pc_gamg_agg_nsmooths refer to?<br></div></div></div></div></div></blockquote><div><br></div></span><div>This is always 1 (its the definition of smoothed aggregation). Mark allows 0 to support unsmoothed aggregation, which may be</div><div>better for easy problems on extremely large machines.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div></div>2) Does increase in the threshold of -pc_gamg_threshold translate to making the coarsening faster?<br></div></div></div></div></blockquote><div><br></div></span><div>Yes, I believe so (easy to check).</div></div></div></div></blockquote></div><br></div></span><div class="gmail_extra">Other way around.</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>