[Swift-user] Transfer directory structure

[Yadu Nand]

Hi Jonathan,

What I'd do in this case is map every file under the directory you want to
send to the compute nodes, into an array and pass
that along to your apps. You can do this mapping using either ext mapper or
array mappers.

I have the following dir structure in my folders:

./dirs/foo_a/foo_a.txt
./dirs/foo_a/foo_b.txt
./dirs/foo_b/bar_1
./dirs/foo_b/bar_2

Here's my ext mapper (mapper.sh) :
#!/bin/bash
find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'

If you are using ext mappers, you would need a script which generates
output in the form [<index>] <filename>
Here I use find to just output files and awk to get the right format.

The swift mapping would be like this:
file array[] <ext ; exec="mapper.sh" >;

You could also use array mappers to read all files you need from a file
containing the filenames. I filled filenames.txt with
the names of all files in the folders.

string[] names = readData("filenames.txt");
file dirmap[] <array_mapper; files=names>;

I've got both cases as examples tarballed here if you'd like to take a look
: http://swift.rcc.uchicago.edu:8042/directory_mapping.tar

Thanks,
Yadu

On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:

> Hello all,
>
> Is there a simple way to specify "all files including files in subfolders
> within a folder" as a file mapper? It looks like the filesys_mapper does
> get everything within a folder, but has problems if there's a folder in
> there.
>
> Jonathan
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>



-- 
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140401/f2ead248/attachment.html>