<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jan 16, 2016, at 4:06 PM, Bhalla, Amneet Pal S <<a href="mailto:amneetb@live.unc.edu" class="">amneetb@live.unc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">
<div class="">Does Instruments save results somewhere (like in a cascade view) that I can send to Barry?<br class="">
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Yes --- "save as..." will save the current trace, and then you can open it back up.</div>
<div><br class="">
</div>
<div>-- Boyce</div>
<div><br class="">
</div>
</div>
<div>
<blockquote type="cite" class="">
<div class="">
<div dir="auto" class="">
<div class="">--Amneet Bhalla<span style="font-size: 13pt;" class=""> </span></div>
<div class=""><br class="">
On Jan 16, 2016, at 1:04 PM, Griffith, Boyce Eugene <<a href="mailto:boyceg@email.unc.edu" class="">boyceg@email.unc.edu</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2016, at 4:00 PM, Bhalla, Amneet Pal S <<a href="mailto:amneetb@live.unc.edu" class="">amneetb@live.unc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">
<div class=""><br class="">
<br class="">
--Amneet Bhalla<span style="font-size: 13pt;" class=""> </span></div>
<div class=""><br class="">
On Jan 16, 2016, at 10:21 AM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class=""><span class=""></span><br class="">
<blockquote type="cite" class=""><span class="">On Jan 16, 2016, at 7:12 AM, Griffith, Boyce Eugene <<a href="mailto:boyceg@email.unc.edu" class="">boyceg@email.unc.edu</a>> wrote:</span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">On Jan 16, 2016, at 12:34 AM, Bhalla, Amneet Pal S <<a href="mailto:amneetb@live.unc.edu" class="">amneetb@live.unc.edu</a>> wrote:</span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">On Jan 15, 2016, at 5:40 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</span><br class="">
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">I am inclined to try</span><br class="">
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">Barry's experiment first, since this may have bugs that we have not yet discovered.</span><br class="">
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">Ok, I tried Barry’s suggestion. The runtime for PetscOptionsFindPair_Private() fell from 14% to mere 1.6%.</span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">If I am getting it right, it’s the petsc options in the KSPSolve() that is sucking up nontrivial amount of time (14 - 1.6)</span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class="">
<blockquote type="cite" class=""><span class="">and not KSPSetFromOptions() itself (1.6%).
</span><br class="">
</blockquote>
</blockquote>
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class="">Barry / Matt / Jed, if we were using KSPReset here and reusing KSPs, would that also bypass these calls to PetscOptionsXXX?</span><br class="">
</blockquote>
<span class=""></span><br class="">
<span class=""> No that is a different issue.</span><br class="">
<span class=""></span><br class="">
<span class=""> In the short term I recommend when running optimized/production you work with a PETSc with those Options checking in KSPSolve commented out, you don't use them anyways*. Since you are using ASM with many subdomains there are many "fast" calls
to KSPSolve which is why for your particular case the the PetscOptionsFindPair_Private takes so much time.</span><br class="">
<span class=""></span><br class="">
<span class=""> Now that you have eliminated this issue I would be very interested in seeing the HPCToolKit or Instruments profiling of the code to see hot spots in the PETSc solver configuration you are using. Thanks
</span><br class="">
</div>
</blockquote>
<div class=""><br class="">
</div>
Barry --- the best way and the least back and forth way would be if I can send you the files (maybe off-list) that you can view in HPCViewer, which is a light weight java script app. You can view which the calling context (which petsc function calls which internal
petsc routine) in a cascade form. If I send you an excel sheet, it would be in a flat view and not that useful for serious profiling.
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Amneet, can you just run with OS X Instruments, which Barry already knows how to use (right Barry?)? :-)</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class="">-- Boyce</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="auto" class="">
<div class=""><br class="">
</div>
<div class="">Let me know if you would like to try that. <br class="">
<blockquote type="cite" class="">
<div class=""><span class=""></span><br class="">
<span class=""> Barry</span><br class="">
<span class=""></span><br class="">
<span class="">* Eventually we'll switch to a KSPPreSolveMonitorSet() and KSPPostSolveMonitorSet() model to eliminate this overhead but still have the functionality.</span><br class="">
<span class=""></span><br class="">
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class="">Thanks,</span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class=""></span><br class="">
</blockquote>
<blockquote type="cite" class=""><span class="">-- Boyce</span><br class="">
</blockquote>
<span class=""></span><br class="">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>