[petsc-dev] Seeking OLCF users complaining about poor build times

[Nathan Collier]

Barry,

No problem.

No packages / master and on /ccs/home/nate

[nate at titan] time python no_packages.py
...
real 41m38.700s
user 1m50.075s
sys 1m30.034s

[nate at titan] time make all
...
real 9m4.977s
user 5m56.686s
sys 28m22.310s

In addition to a much slower configure time, the home areas are not
accessible to batch submissions. So without the reconfigure script, I could
not do a full build from home.

Nate


On Thu, Feb 26, 2015 at 11:19 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  Nathan,
>
>     Can you please do a configure make on your home directory system?
>
>      It could be since Lustre is a parallel filesystem designed for high
> bandwidth IO from a parallel computer it is just bad for compiles etc.
>
>   Barry
>
>
>
> https://www.olcf.ornl.gov/support/getting-started/
>
> home areas
> Home areas/directories are provided on a Network File System (NFS). The
> user home areas are by default accessible only to the owning user.
> Similarly the project home areas are accessible by only member of the
> project. The areas are backed-up but available space is limited. Because
> space is limited, each area has a quota. Users should store small source
> code, scripts, and other similar items in the area. Users should not store
> large job output or input in the area. Job I/O should be performed in the
> system’s temporary work area.
> work areas
> Temporary work directories are provided to each user and project on Lustre
> file systems. Similar to the home areas, by default, the user and project
> areas are accessible to the user and project members. The areas provide a
> large amount of storage, but are not backed-up. In general, job I/O
> performance will be faster in the lustre areas than the NFS mounted home
> areas. The temporary work areas are regularly purged of data that has not
> been recently accessed, because of this all needed data should be backed-up
> to the HPSS.
>
>
> > On Feb 26, 2015, at 10:00 PM, Nathan Collier <
> nathaniel.collier at gmail.com> wrote:
> >
> > > They never told you to do compiles on particular file systems or
> anything?  For example did they ever say DON'T compile on the lustre file
> system?
> >
> > I have not received any guidance about where to or not to compile. A
> quick perusing of the user guide doesn't make that clear to me. If it isn't
> already obvious, I am far from a Titan expert :)
> >
> > Do you know someone here at ORNL who I can talk to to make sure I am not
> missing something silly?
> >
> > > Perhaps the 52m 'sys' time could be luster overhead
> >
> > It could be, I have heard a lot of complaints about this from people who
> need to write output or read from files in their applications.
> >
> > Nate
> >
> >
> > On Thu, Feb 26, 2015 at 10:36 PM, Satish Balay <balay at mcs.anl.gov>
> wrote:
> > On Thu, 26 Feb 2015, Barry Smith wrote:
> >
> > >
> > > > On Feb 26, 2015, at 9:16 PM, Nathan Collier <
> nathaniel.collier at gmail.com> wrote:
> > > >
> > > > Barry,
> > > >
> > > > > Do you know where /tmp is on the system? Presumably it is fast?
> > > >
> > > > I am not sure about that. There is a /tmp but I am not sure what it
> is or if/how I can use it.
> > >
> > >    They never told you to do compiles on particular file systems or
> anything?  For example did they ever say DON'T compile on the lustre file
> system?
> >
> > Ah - I misunderstood earlier. You'd like to have a petsc clone in /tmp
> > - and do the build there - and compare the difference [with the
> > current luster build].
> >
> > Perhaps the 52m 'sys' time could be luster overhead - and that woud
> > disappear with a petsc build in /tmp. [and could be much faster].
> >
> > Satish
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150227/1ff85cdd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: no_packages_home.log
Type: application/octet-stream
Size: 2603716 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150227/1ff85cdd/attachment.obj>