[Swift-user] handling prestaged data on multiple sites

[Justin M Wozniak]

Collective Data Management.  The general idea is to exploit user 
information to improve data access for Swift scripts.  Specifically, you 
can plug in new data access mechanisms without modifying the application 
script- most of the details are specified in a small user configuration 
file (analogous to tc.data) and managed by extensions to the wrapper 
script (_swiftwrap, etc.).  Dynamically renaming data sources is a big 
part of it.

The framework is there in trunk and a few methods are implemented, 
including a staging-skipping DIRECT method, a BROADCAST, and an 
output-aggregating GATHER method.  I can easily put in a new method for 
Globus.org, I'll take a look and get back to you, Allan.

On Tue, 4 May 2010, Ian Foster wrote:

> What is CDM?
>
> Ian
>
> On May 4, 2010, at 10:45 AM, Allan Espinosa wrote:
>
>> Hi Justin,
>>
>> I used globus.org's data movement service.
>>
>> Right now I have the dataset on these OSG sites:
>>
>> ff-grid.unl.edu:/panfs/panasas/CMS/data/scec/RuptureVariations/*
>> tuscany.med.harvard.edu:/osg/storage/data/engage/scec/RuptureVariations/*
>>
>> On Mar, 2010-05-04 at 10:20 -0500, Justin Wozniak wrote:
>>>
>>> This is probably a good use case for CDM.  How did you prestage the
>>> files?
>>>
>>>
>>>
>>> ______________________________________________________________________
>>> Allan Espinosa wrote:
>>>
>>> Hi,
>>>
>>> I prestaged 11k small files (total ~2GB) to two sites. if I were just
>>> computing on 1 site, i can just place hardwired strings of absolute
>>> pathnames to disable staging. I was originally thinking of passing
>>> '$OSG_DATA' variable names since the app functions wraps to bash
>>> shell
>>> scripts. this backfired because swift passes these string args
>>> literally.
>>>
>>> any other recommended approaches?
>>>
>>> thanks!
>>> -Allan
>>
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>

-- 
Justin M Wozniak