[Swift-user] Transfer directory structure

[David Kelly]

I created a ticket about this on Monday because I was running into similar
issues in my scripts. An ext mapper worked for me, but I think this is a
common pattern we can make easier in future releases.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.


On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:

> Hi Jonathan,
>
> What I'd do in this case is map every file under the directory you want to
> send to the compute nodes, into an array and pass
> that along to your apps. You can do this mapping using either ext mapper
> or array mappers.
>
> I have the following dir structure in my folders:
>
> ./dirs/foo_a/foo_a.txt
> ./dirs/foo_a/foo_b.txt
> ./dirs/foo_b/bar_1
> ./dirs/foo_b/bar_2
>
> Here's my ext mapper (mapper.sh) :
> #!/bin/bash
> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>
> If you are using ext mappers, you would need a script which generates
> output in the form [<index>] <filename>
> Here I use find to just output files and awk to get the right format.
>
> The swift mapping would be like this:
> file array[] <ext ; exec="mapper.sh" >;
>
> You could also use array mappers to read all files you need from a file
> containing the filenames. I filled filenames.txt with
> the names of all files in the folders.
>
> string[] names = readData("filenames.txt");
> file dirmap[] <array_mapper; files=names>;
>
> I've got both cases as examples tarballed here if you'd like to take a
> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>
> Thanks,
> Yadu
>
> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>
>> Hello all,
>>
>> Is there a simple way to specify "all files including files in subfolders
>> within a folder" as a file mapper? It looks like the filesys_mapper does
>> get everything within a folder, but has problems if there's a folder in
>> there.
>>
>> Jonathan
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>
>
>
> --
> Yadu Nand B
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140401/8f17f450/attachment.html>