[Swift-devel] Use case and examples needed to avoid large directories

[andrewj at uchicago.edu]

Thanks for the input.


  First off, before I attempt to pose a further question I
have, who is in the CI currently or regularly?  I think it
would be more efficient for me to meet up and possibly discuss
some of this issues face to face as I try to type out these
commands.  Let me know which days are best to stop in.

Ok, quesiton is will this system allow for me to simply lay
out the input files in a certain directory structure (possibly
mulitple paths deep) and reference this data in one or two
lines of code in Swift without explicitly outlining locations
for the data.  

For instance,  say I have 10,000 input files total.  And it is
split into two types, malignant and benign, 5000 each.  But
these guys are distributed in a mulit-layer directory tree of
some sort.

like ~/swifthome/malignant/a01/b01/2/*
     ~/swifthome/benign/c01/h5/9/*

what would I write to define an array of inputs (malignant and
benign separately) for all the files included in these
subdirectories?

Thanks,
Andrew   



---- Original message ----
>Date: Sat, 29 Sep 2007 08:42:59 +0000 (GMT)
>From: Ben Clifford <benc at hawaga.org.uk>  
>Subject: Re: [Swift-devel] Use case and examples needed to
avoid large directories  
>To: Michael Wilde <wilde at mcs.anl.gov>
>Cc: swift-devel <swift-devel at ci.uchicago.edu>, Andrew
Jamieson <andrewj at uchicago.edu>
>
>In brief, if you have a file in your submit directory called:
>
>  (swifthome)/a/b.txt
>
>and map it somehow:
>
> file f <"a/b.txt">;
>
>then on the submit side, in the run directory it will map to
individual 
>job execution directory like this:
>
>  (individualjobdir)/a/b.txt
>
>This should be the case for inputs, outputs and intermediates.
>
>If you want to map everything in the 'a' directory, you can
map something 
>like:
>
> file f[] <filesys_mapper; location="a">;
>
>to get an array containing every file in a.
>
>
>Its meant to be the case that you can also store stuff
elsewhere (on the 
>same machine or on any gridftp server) rather than directly
in the submit 
>directory but I had some troubles when playing with it so I
have a bug 
>open and am waiting on that before further documentation of
that bit.
>
>--