[petsc-dev] [Ideas-team] Seeking OLCF users complaining about poor build times

[Barry Smith]

  Same text also in the attachment.

   Barry

David,

    Nathan Collier has kindly run a test on Titan, Satish on Mira and Hopper, and Victor on Ranger with a basic optimized build of PETSc (all C code)

    Please find below some configure and make timings from the latest PETSc master.

     The Titan times for both configure and make are unacceptable. For total build time Titan is 3.5 times slower than Mira and Hopper and at least 10 times slower than laptops. The "time" results on Titan are disturbing

configure 
real	14m32.169s   (since the user + sys time is much less than real time, what is it waiting on?)
user	1m51.527s
sys	3m40.734s

make
real	15m56.004s
user	8m8.971s
sys	52m42.734s  (why so much?)

which I read as either the filesystem or the compiler system (location of the compilers, license server of the compilers, ...) is really badly configured.

   The Hopper configure time with the default TMPDIR=/scratch/scratchdirs/balay is is unacceptable but if you actually use the real /tmp it becomes somewhat reasonable.

Feel free to share this information with local experts,

  


I suggest you view the below table in a fixed width font editor like Emacs or Vi so the columns line up.

                    configure time    make time   Total      compilers     filesystem

Titan                14m32s         15m56s        30m28s      Intel 14    /lustre/atlas1/geo103/proj-shared/
                     41m38s          9m5s         50m43s                 /ccs/home/  (no load on login node)
		     13m                                                             (no load on a different login node)

Mira                  6m59s          1m49s         8m48s       IBM        /gpfs/mira-home/

Hopper               23m17           1m45s        25m2s                   /global/u2/b/balay/petsc.clone default TMPDIR=/scratch/scratchdirs/balay
		      6m17s          1m39s         7m57s                   manually set TMPDIR=/tmp

NSF Ranger UT Austin  5m10s          1m28s         6m38s                     default, whatever it is

Linux laptop            53s          1m13s         2m6s         Gnu           compile and compiler local

Apple laptop          1m14s            54s         2m8s         clang         compile and compiler local

Linux workstation     1m11s            22s         1m33s        Gnu         compile and compiler local
                      1m37s            29s         2m6s         Gnu         compile directory local; compiler directory remote
                      3m11s            25s         3m36s       Intel 13     compile directory local; compiler directory remote

PETSc has about 1000 source files that need compiling

The configure is essentially sequential, the make extremely parallel.

During configure the source code is on the listed file system, all .o and executables  are on /tmp

During the make the source code and all .o are on the listed file system


> On Feb 25, 2015, at 11:23 AM, David E. Bernholdt <bernholdtde at ornl.gov> wrote:
> 
> At the kick-off meetings, one of the general complaints I heard
> expressed about the facilities was the slow build times compared to
> personal systems.
> 
> If you have this complaint and are an OLCF user, and are willing to work
> with us a little to try to understand your experience in more detail,
> please contact me (individually, not reply-all).
> 
> This is a facility thing, not an IDEAS thing, so I can't speak for the
> other facilities.  But we've recently received some other similar
> comments, and we're trying to dig into what's happening.
> 
> Thanks
> -- 
> David E. Bernholdt                | Email: bernholdtde at ornl.gov
> Oak Ridge National Laboratory     | Phone: +1 865-574-3147
> http://www.csm.ornl.gov/~bernhold | Fax:   +1 865-576-5491
> _______________________________________________
> Ideas-team mailing list
> Ideas-team at lists.mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/ideas-team
-------------- next part --------------
A non-text attachment was scrubbed...
Name: timings
Type: application/octet-stream
Size: 2884 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150227/405c1922/attachment.obj>