[petsc-dev] Seeking OLCF users complaining about poor build times

[Barry Smith]

 Nathan,

    Can you please do a configure make on your home directory system? 

     It could be since Lustre is a parallel filesystem designed for high bandwidth IO from a parallel computer it is just bad for compiles etc.

  Barry



https://www.olcf.ornl.gov/support/getting-started/

home areas
Home areas/directories are provided on a Network File System (NFS). The user home areas are by default accessible only to the owning user. Similarly the project home areas are accessible by only member of the project. The areas are backed-up but available space is limited. Because space is limited, each area has a quota. Users should store small source code, scripts, and other similar items in the area. Users should not store large job output or input in the area. Job I/O should be performed in the system’s temporary work area.
work areas
Temporary work directories are provided to each user and project on Lustre file systems. Similar to the home areas, by default, the user and project areas are accessible to the user and project members. The areas provide a large amount of storage, but are not backed-up. In general, job I/O performance will be faster in the lustre areas than the NFS mounted home areas. The temporary work areas are regularly purged of data that has not been recently accessed, because of this all needed data should be backed-up to the HPSS.


> On Feb 26, 2015, at 10:00 PM, Nathan Collier <nathaniel.collier at gmail.com> wrote:
> 
> > They never told you to do compiles on particular file systems or anything?  For example did they ever say DON'T compile on the lustre file system?
> 
> I have not received any guidance about where to or not to compile. A quick perusing of the user guide doesn't make that clear to me. If it isn't already obvious, I am far from a Titan expert :) 
> 
> Do you know someone here at ORNL who I can talk to to make sure I am not missing something silly?
> 
> > Perhaps the 52m 'sys' time could be luster overhead 
> 
> It could be, I have heard a lot of complaints about this from people who need to write output or read from files in their applications.
> 
> Nate
> 
> 
> On Thu, Feb 26, 2015 at 10:36 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> On Thu, 26 Feb 2015, Barry Smith wrote:
> 
> >
> > > On Feb 26, 2015, at 9:16 PM, Nathan Collier <nathaniel.collier at gmail.com> wrote:
> > >
> > > Barry,
> > >
> > > > Do you know where /tmp is on the system? Presumably it is fast?
> > >
> > > I am not sure about that. There is a /tmp but I am not sure what it is or if/how I can use it.
> >
> >    They never told you to do compiles on particular file systems or anything?  For example did they ever say DON'T compile on the lustre file system?
> 
> Ah - I misunderstood earlier. You'd like to have a petsc clone in /tmp
> - and do the build there - and compare the difference [with the
> current luster build].
> 
> Perhaps the 52m 'sys' time could be luster overhead - and that woud
> disappear with a petsc build in /tmp. [and could be much faster].
> 
> Satish
>