[Swift-user] Question about using ramdisk

[Michael Wilde]

Dear Lorenzo,

We'll need a bit more information on your data flow pattern t help you with this.

In general, the strategy is to move as much IO from shared filesystems to filesystems that are local to each node.

If your data flow pattern is one on which you create many independent datasets on the nodes that eventually need to read by a single app invocation (e.g. one that needs to reduce or summarize all the outputs). Im assuming this is what you meant by "how to tell swift that this is where it should pick up the output files from the run"?

One way to do this is to use provider staging, which will send the output files back to the submit host. Another strategy is to use CDM "direct" mode to *efficiently* write output data to a shared filesystem (eg lustre) location  A third way - not yet well tested - is to use CDM "gather" mode to collect the output files into tar archives, one archive per node, which will reduce the number of files that need to be written to a shared location.

Rather than trying to learn what youre trying to do with a long thread on the swift-user list, lets switch to swift-support or an in-person meeting, and then report the final suggestions back to swift-user and document them in the User Guide.

- Mike

----- Original Message -----
> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Friday, September 7, 2012 7:50:44 PM
> Subject: [Swift-user] Question about using ramdisk
> Dear all,
> 
> I just realized that I locked myself into a design corner.
> 
> We use Matlab executables run by a swift wrapper.
> 
> I wanted to use ramdisk (/dev/shm or whatever works on each node)
> because our lustre filesystem isn't liking too much to be poked all
> the time and it is making sure we know that. I planned to move there.
> I just realized that I have no idea how to tell swift that this is
> where it should pick up the output files from the run.
> 
> The easy fix is to simply add a mv statement (or whatever works best,
> we'll have to try) to the wrapper scripts, since while they will have
> a different source (the local /dev/shm) they have the same target
> (/lustre), but I was wondering if Swift has a direct way to handle
> that. I am asking because the script will be asked to run on different
> systems where swift will know where it was started and where the
> wrappers are being run, the same is not true for the wrappers.
> 
> Any ideas?
> 
> Thanks a lot,
> 
> Lorenzo
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory