<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Feb 27, 2015 at 10:29 AM, Todd Gamblin <span dir="ltr"><<a href="mailto:tgamblin@llnl.gov" target="_blank">tgamblin@llnl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Barry:<br>
<br>
I remember that ALCF attempted to address this problem at one point or<br>
another with "tmpicc" compiler wrappers. As I remember the idea was that<br>
they stored the compiler's tmp files in some local storage on the login<br>
node. I think that was back when ANL's main machine was Intrepid, and I<br>
don't know where those compilers went on Mira. Do you remember this?<br>
<br>
In general I'm not sure that just moving the compiler temp files is going<br>
to cut it. I think you really want to do the build out of /tmp or some<br>
other filesystem. Spack does this automatically for its builds -- on LLNL<br>
machines I build much faster by just finding the local tmp space and using<br>
it for all the builds. Spack is also able to put the entire build out in<br>
tmp space, because you just tell it the software name, and it handles the<br>
details of where it is downloaded and expanded. It's not perfect, because<br>
it looks at $TMP, $TMPDIR, and some other LLNL-specific places.<br></blockquote><div><br></div><div>We already do this in configure.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If it turns out that configuring NFS (or in ANL's case, I think it's GPFS)<br>
to be fast on a set of loaded login nodes is not feasible, it might be<br>
nice to have some kind of recommendations for build staging.<br>
<span class="HOEnZb"><font color="#888888"><br>
-Todd<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 2/27/15, 8:09 AM, "David E. Bernholdt" <<a href="mailto:bernholdtde@ornl.gov">bernholdtde@ornl.gov</a>> wrote:<br>
<br>
>Barry, thanks, this is extremely helpful. I'll have the OLCF folks<br>
>contact Nathan if they need any further info or have other experiments<br>
>to try.<br>
><br>
>On 02/27/2015 11:03 AM, Barry Smith wrote:<br>
>><br>
>> Same text also in the attachment.<br>
>><br>
>> Barry<br>
>><br>
>> David,<br>
>><br>
>> Nathan Collier has kindly run a test on Titan, Satish on Mira and<br>
>>Hopper, and Victor on Ranger with a basic optimized build of PETSc (all<br>
>>C code)<br>
>><br>
>> Please find below some configure and make timings from the latest<br>
>>PETSc master.<br>
>><br>
>> The Titan times for both configure and make are unacceptable. For<br>
>>total build time Titan is 3.5 times slower than Mira and Hopper and at<br>
>>least 10 times slower than laptops. The "time" results on Titan are<br>
>>disturbing<br>
>><br>
>> configure<br>
>> real 14m32.169s (since the user + sys time is much less than real<br>
>>time, what is it waiting on?)<br>
>> user 1m51.527s<br>
>> sys 3m40.734s<br>
>><br>
>> make<br>
>> real 15m56.004s<br>
>> user 8m8.971s<br>
>> sys 52m42.734s (why so much?)<br>
>><br>
>> which I read as either the filesystem or the compiler system (location<br>
>>of the compilers, license server of the compilers, ...) is really badly<br>
>>configured.<br>
>><br>
>> The Hopper configure time with the default<br>
>>TMPDIR=/scratch/scratchdirs/balay is is unacceptable but if you actually<br>
>>use the real /tmp it becomes somewhat reasonable.<br>
>><br>
>> Feel free to share this information with local experts,<br>
>><br>
>><br>
>><br>
>><br>
>> I suggest you view the below table in a fixed width font editor like<br>
>>Emacs or Vi so the columns line up.<br>
>><br>
>> configure time make time Total compilers<br>
>> filesystem<br>
>><br>
>> Titan 14m32s 15m56s 30m28s Intel 14<br>
>> /lustre/atlas1/geo103/proj-shared/<br>
>> 41m38s 9m5s 50m43s<br>
>> /ccs/home/ (no load on login node)<br>
>> 13m<br>
>>(no load on a different login node)<br>
>><br>
>> Mira 6m59s 1m49s 8m48s IBM<br>
>> /gpfs/mira-home/<br>
>><br>
>> Hopper 23m17 1m45s 25m2s<br>
>> /global/u2/b/balay/petsc.clone default<br>
>>TMPDIR=/scratch/scratchdirs/balay<br>
>> 6m17s 1m39s 7m57s manually<br>
>>set TMPDIR=/tmp<br>
>><br>
>> NSF Ranger UT Austin 5m10s 1m28s 6m38s<br>
>> default, whatever it is<br>
>><br>
>> Linux laptop 53s 1m13s 2m6s Gnu<br>
>> compile and compiler local<br>
>><br>
>> Apple laptop 1m14s 54s 2m8s clang<br>
>> compile and compiler local<br>
>><br>
>> Linux workstation 1m11s 22s 1m33s Gnu<br>
>> compile and compiler local<br>
>> 1m37s 29s 2m6s Gnu<br>
>> compile directory local; compiler directory remote<br>
>> 3m11s 25s 3m36s Intel 13<br>
>> compile directory local; compiler directory remote<br>
>><br>
>> PETSc has about 1000 source files that need compiling<br>
>><br>
>> The configure is essentially sequential, the make extremely parallel.<br>
>><br>
>> During configure the source code is on the listed file system, all .o<br>
>>and executables are on /tmp<br>
>><br>
>> During the make the source code and all .o are on the listed file system<br>
>><br>
>><br>
>>> On Feb 25, 2015, at 11:23 AM, David E. Bernholdt<br>
>>><<a href="mailto:bernholdtde@ornl.gov">bernholdtde@ornl.gov</a>> wrote:<br>
>>><br>
>>> At the kick-off meetings, one of the general complaints I heard<br>
>>> expressed about the facilities was the slow build times compared to<br>
>>> personal systems.<br>
>>><br>
>>> If you have this complaint and are an OLCF user, and are willing to<br>
>>>work<br>
>>> with us a little to try to understand your experience in more detail,<br>
>>> please contact me (individually, not reply-all).<br>
>>><br>
>>> This is a facility thing, not an IDEAS thing, so I can't speak for the<br>
>>> other facilities. But we've recently received some other similar<br>
>>> comments, and we're trying to dig into what's happening.<br>
>>><br>
>>> Thanks<br>
>>> --<br>
>>> David E. Bernholdt | Email: <a href="mailto:bernholdtde@ornl.gov">bernholdtde@ornl.gov</a><br>
>>> Oak Ridge National Laboratory | Phone: <a href="tel:%2B1%20865-574-3147" value="+18655743147">+1 865-574-3147</a><br>
>>> <a href="http://www.csm.ornl.gov/~bernhold" target="_blank">http://www.csm.ornl.gov/~bernhold</a> | Fax: <a href="tel:%2B1%20865-576-5491" value="+18655765491">+1 865-576-5491</a><br>
>>> _______________________________________________<br>
>>> Ideas-team mailing list<br>
>>> <a href="mailto:Ideas-team@lists.mcs.anl.gov">Ideas-team@lists.mcs.anl.gov</a><br>
>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/ideas-team" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/ideas-team</a><br>
><br>
><br>
>--<br>
>David E. Bernholdt | Email: <a href="mailto:bernholdtde@ornl.gov">bernholdtde@ornl.gov</a><br>
>Oak Ridge National Laboratory | Phone: <a href="tel:%2B1%20865-574-3147" value="+18655743147">+1 865-574-3147</a><br>
><a href="http://www.csm.ornl.gov/~bernhold" target="_blank">http://www.csm.ornl.gov/~bernhold</a> | Fax: <a href="tel:%2B1%20865-576-5491" value="+18655765491">+1 865-576-5491</a><br>
>_______________________________________________<br>
>Ideas-team mailing list<br>
><a href="mailto:Ideas-team@lists.mcs.anl.gov">Ideas-team@lists.mcs.anl.gov</a><br>
><a href="https://lists.mcs.anl.gov/mailman/listinfo/ideas-team" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/ideas-team</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>