[Swift-user] Transfer directory structure

[Mihael Hategan]

At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
you should be able to say:

file[] f <FilesysMapper; location=".", pattern="**/*.*">;

Mihael

On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> I created a ticket about this on Monday because I was running into similar
> issues in my scripts. An ext mapper worked for me, but I think this is a
> common pattern we can make easier in future releases.
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> 
> 
> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
> 
> > Hi Jonathan,
> >
> > What I'd do in this case is map every file under the directory you want to
> > send to the compute nodes, into an array and pass
> > that along to your apps. You can do this mapping using either ext mapper
> > or array mappers.
> >
> > I have the following dir structure in my folders:
> >
> > ./dirs/foo_a/foo_a.txt
> > ./dirs/foo_a/foo_b.txt
> > ./dirs/foo_b/bar_1
> > ./dirs/foo_b/bar_2
> >
> > Here's my ext mapper (mapper.sh) :
> > #!/bin/bash
> > find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >
> > If you are using ext mappers, you would need a script which generates
> > output in the form [<index>] <filename>
> > Here I use find to just output files and awk to get the right format.
> >
> > The swift mapping would be like this:
> > file array[] <ext ; exec="mapper.sh" >;
> >
> > You could also use array mappers to read all files you need from a file
> > containing the filenames. I filled filenames.txt with
> > the names of all files in the folders.
> >
> > string[] names = readData("filenames.txt");
> > file dirmap[] <array_mapper; files=names>;
> >
> > I've got both cases as examples tarballed here if you'd like to take a
> > look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >
> > Thanks,
> > Yadu
> >
> > On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >
> >> Hello all,
> >>
> >> Is there a simple way to specify "all files including files in subfolders
> >> within a folder" as a file mapper? It looks like the filesys_mapper does
> >> get everything within a folder, but has problems if there's a folder in
> >> there.
> >>
> >> Jonathan
> >>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>
> >
> >
> >
> > --
> > Yadu Nand B
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user