[Swift-devel] Fwd: [Users] Appropriate use of $HOME

[Michael Wilde]

This note is very relevant to Swift...

We need to show, in the Users Guide, good practices for filesystem usage.

- Mike

----- Forwarded Message -----
From: "David Turner" <dpturner at lbl.gov>
To: users at nersc.gov
Sent: Thursday, December 12, 2013 1:51:09 PM
Subject: [Users] Appropriate use of $HOME

Dear NERSC User,

We have continued to receive reports of sluggish response
for common interactive operations such as ls, df, vi, and
others.  Occasionally, user logins also take longer than
expected.  The purpose of this email to suggest how all
users can help us solve these issues.

Investigations consistently show two primary causes for this
performance degradation:  batch jobs performing significant
I/O in global home directories, and parallel file transfer
activities in global homes.

NERSC's various file systems are tuned for different use cases.
The home file systems are tuned for efficient small-file and
metadata operations, as are typically encountered when building
applications.  Conversely, the various scratch and project file
systems are tuned for high-volume streaming I/O operations.

Please do not run batch jobs in your home directory.  Instead,
run batch jobs only from a scratch directory or a project
directory.  Also, even if running in a scratch or a project
directory, try to avoid any explicit I/O to/from your home
directory.

Similarly, if you need to transfer data into or out of your
home directory, please use no more than two simultaneous data
transfer clients.  This applies to multiple instances of ftp,
hsi, htar, and cp and/or scp.  Note that running multiple
instances of these tools can not only put a heavy load on
the file system, they can also put a heavy load on the login
nodes themselves, contributing to sluggish interactive response.

Lastly, file system performance can suffer when even a few
active users are close to their disk quota limits.  Everyone
can benefit if you maintain sufficient "headroom" between
your usage in $HOME and your 40GB quota.  You can use the
"myquota" command to view your disk usage and quota values.
Effective use of HPSS can help you manage your data storage needs.

If you have any questions about these issues, please contact
NERSC Consulting at:

   1-800-666-3772, menu option 3, 8 am - 5 pm, Pacific time
   (510) 486-8600, menu option 3, 8 am - 5 pm, Pacific time
   consult at nersc.gov
   http://help.nersc.gov/

Thank you for your cooperation in helping make NERSC compute
and data systems responsive for all users.

-- 
Best regards,

David Turner
User Services Group        email: dpturner at lbl.gov
NERSC Division             phone: (510) 486-4027
Lawrence Berkeley Lab        fax: (510) 486-4316
_______________________________________________
Users mailing list
Users at nersc.gov