[Swift-user] Data transfer error

Mihael Hategan hategan at mcs.anl.gov
Fri May 23 15:47:41 CDT 2014


On Fri, 2014-05-23 at 20:33 +0000, Bronevetsky, Greg wrote:
> Yes, at least as long as it is cleaned out at the end of each job so that I don't get memory leaks across jobs.

First the copy not link part:
 * leave <workdirectory> as it is in sites.xml, but add something like:
<scratch>/tmp/swift</scratch>.
 
This will cause swift to use /tmp/swift to run jobs, and it will copy
fies to it and back as necessary.

Alternatively, you should be able to avoid hitting the shared FS almost
entirely (almost because the PBS jobs might still use it for scripts and
log files).:

 * set "use.provider.staging=true" in swift.properties
 * <workdirectory>/tmp/swift</workdirectory> in sites.xml
 * add <profile namespace="swift" key="stagingMethod">file</profile> to
sites.xml
 * compile (or unpack) swift on a local disk on the head node
 * also have the directories where you run swift and data files on a
local disk on the head node

Mihael




More information about the Swift-user mailing list