[Swift-user] Optimizing Swift overhead of file copy - reg.

[Yadu Nand]

Hi Karthikeyan,

I'm guessing from the your mention of "processing from their original
location" that Mike
must have been referring to staging mode direct.

In this mode, swift points your applications to your data files which are
on a shared filesystem
and avoid making copies. There are cases in which application access
patterns are not
suitable for a shared filesystem, in such cases you can have swift make a
copy from
the shared-filesystem to a local disk on the node. Since local disks are
generally faster
than shared-filesystems, this might make more sense if your application is
I/O intensive.

Here's a document that might help give you a clearer picture :
https://github.com/swift-lang/swift-k/blob/master/docs/staging/staging.asc

Thanks,
Yadu







On Wed, Feb 11, 2015 at 11:34 AM, Karthikeyan Balasubramanian <
karthikeyanb at uchicago.edu> wrote:

>  Hi,
>
>  When jobs are run in Swift, it copies the data from the original
> location to the work folder for processing. Since our data are in the order
> of 15-30 GB, it adds unwanted overhead for the process. I recall from my
> early discussions with Michael Wilde that this could be optimized so that
> the files can be processed from their original location.
>
>  I am wondering as how should I proceed to achieve this optimization.
> Your inputs are greatly appreciated.
>
>  Thanks.
> B.K.
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>



-- 
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20150211/48fece8b/attachment.html>