[Swift-user] handling prestaged data on multiple sites

[Ian Foster]

PS: If we want to do wide area stuff with CDM, it should be invoking Globus.org.

On May 4, 2010, at 1:25 PM, Justin M Wozniak wrote:

> 
> Collective Data Management.  The general idea is to exploit user information to improve data access for Swift scripts.  Specifically, you can plug in new data access mechanisms without modifying the application script- most of the details are specified in a small user configuration file (analogous to tc.data) and managed by extensions to the wrapper script (_swiftwrap, etc.).  Dynamically renaming data sources is a big part of it.
> 
> The framework is there in trunk and a few methods are implemented, including a staging-skipping DIRECT method, a BROADCAST, and an output-aggregating GATHER method.  I can easily put in a new method for Globus.org, I'll take a look and get back to you, Allan.
> 
> On Tue, 4 May 2010, Ian Foster wrote:
> 
>> What is CDM?
>> 
>> Ian
>> 
>> On May 4, 2010, at 10:45 AM, Allan Espinosa wrote:
>> 
>>> Hi Justin,
>>> 
>>> I used globus.org's data movement service.
>>> 
>>> Right now I have the dataset on these OSG sites:
>>> 
>>> ff-grid.unl.edu:/panfs/panasas/CMS/data/scec/RuptureVariations/*
>>> tuscany.med.harvard.edu:/osg/storage/data/engage/scec/RuptureVariations/*
>>> 
>>> On Mar, 2010-05-04 at 10:20 -0500, Justin Wozniak wrote:
>>>> 
>>>> This is probably a good use case for CDM.  How did you prestage the
>>>> files?
>>>> 
>>>> 
>>>> 
>>>> ______________________________________________________________________
>>>> Allan Espinosa wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I prestaged 11k small files (total ~2GB) to two sites. if I were just
>>>> computing on 1 site, i can just place hardwired strings of absolute
>>>> pathnames to disable staging. I was originally thinking of passing
>>>> '$OSG_DATA' variable names since the app functions wraps to bash
>>>> shell
>>>> scripts. this backfired because swift passes these string args
>>>> literally.
>>>> 
>>>> any other recommended approaches?
>>>> 
>>>> thanks!
>>>> -Allan
>>> 
>>> 
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>> 
> 
> -- 
> Justin M Wozniak