[Swift-user] Transfer directory structure

[Jonathan Ozik]

Yadu, David, Mihael,

Thanks for your responses.
I'm thinking of using the ext mapper for now. Would the trunk/0.95 be available on Midway?

Jonathan

On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> you should be able to say:
> 
> file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> 
> Mihael
> 
> On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
>> I created a ticket about this on Monday because I was running into similar
>> issues in my scripts. An ext mapper worked for me, but I think this is a
>> common pattern we can make easier in future releases.
>> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
>> 
>> 
>> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
>> 
>>> Hi Jonathan,
>>> 
>>> What I'd do in this case is map every file under the directory you want to
>>> send to the compute nodes, into an array and pass
>>> that along to your apps. You can do this mapping using either ext mapper
>>> or array mappers.
>>> 
>>> I have the following dir structure in my folders:
>>> 
>>> ./dirs/foo_a/foo_a.txt
>>> ./dirs/foo_a/foo_b.txt
>>> ./dirs/foo_b/bar_1
>>> ./dirs/foo_b/bar_2
>>> 
>>> Here's my ext mapper (mapper.sh) :
>>> #!/bin/bash
>>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>>> 
>>> If you are using ext mappers, you would need a script which generates
>>> output in the form [<index>] <filename>
>>> Here I use find to just output files and awk to get the right format.
>>> 
>>> The swift mapping would be like this:
>>> file array[] <ext ; exec="mapper.sh" >;
>>> 
>>> You could also use array mappers to read all files you need from a file
>>> containing the filenames. I filled filenames.txt with
>>> the names of all files in the folders.
>>> 
>>> string[] names = readData("filenames.txt");
>>> file dirmap[] <array_mapper; files=names>;
>>> 
>>> I've got both cases as examples tarballed here if you'd like to take a
>>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>>> 
>>> Thanks,
>>> Yadu
>>> 
>>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>>> 
>>>> Hello all,
>>>> 
>>>> Is there a simple way to specify "all files including files in subfolders
>>>> within a folder" as a file mapper? It looks like the filesys_mapper does
>>>> get everything within a folder, but has problems if there's a folder in
>>>> there.
>>>> 
>>>> Jonathan
>>>> 
>>>> _______________________________________________
>>>> Swift-user mailing list
>>>> Swift-user at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Yadu Nand B
>>> 
>>> 
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
>