[Swift-user] Transfer directory structure

[Jonathan Ozik]

Thanks David (especially for pointing me to the newer user guide).
Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that accepts the extended glob patterns?

Jonathan

On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:

> Hi Jonathan,
> 
> A fairly recent version of 0.95 is available on Midway in the module "swift/0.95-RC5". It should backwards compatible with 0.94. You may see some warnings about deprecated use of @ in front of functions, and you will see logs going into a run directory named run001, run002, etc instead of going to your current working directory. There's more information about this and the other new (and optional) configuration changes at http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration. Please let me know if you have any issues.
> 
> Thanks,
> David
> 
> 
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> Yadu, David, Mihael,
> 
> Thanks for your responses.
> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be available on Midway?
> 
> Jonathan
> 
> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > you should be able to say:
> >
> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> >
> > Mihael
> >
> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> >> I created a ticket about this on Monday because I was running into similar
> >> issues in my scripts. An ext mapper worked for me, but I think this is a
> >> common pattern we can make easier in future releases.
> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> >>
> >>
> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
> >>
> >>> Hi Jonathan,
> >>>
> >>> What I'd do in this case is map every file under the directory you want to
> >>> send to the compute nodes, into an array and pass
> >>> that along to your apps. You can do this mapping using either ext mapper
> >>> or array mappers.
> >>>
> >>> I have the following dir structure in my folders:
> >>>
> >>> ./dirs/foo_a/foo_a.txt
> >>> ./dirs/foo_a/foo_b.txt
> >>> ./dirs/foo_b/bar_1
> >>> ./dirs/foo_b/bar_2
> >>>
> >>> Here's my ext mapper (mapper.sh) :
> >>> #!/bin/bash
> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >>>
> >>> If you are using ext mappers, you would need a script which generates
> >>> output in the form [<index>] <filename>
> >>> Here I use find to just output files and awk to get the right format.
> >>>
> >>> The swift mapping would be like this:
> >>> file array[] <ext ; exec="mapper.sh" >;
> >>>
> >>> You could also use array mappers to read all files you need from a file
> >>> containing the filenames. I filled filenames.txt with
> >>> the names of all files in the folders.
> >>>
> >>> string[] names = readData("filenames.txt");
> >>> file dirmap[] <array_mapper; files=names>;
> >>>
> >>> I've got both cases as examples tarballed here if you'd like to take a
> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >>>
> >>> Thanks,
> >>> Yadu
> >>>
> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >>>
> >>>> Hello all,
> >>>>
> >>>> Is there a simple way to specify "all files including files in subfolders
> >>>> within a folder" as a file mapper? It looks like the filesys_mapper does
> >>>> get everything within a folder, but has problems if there's a folder in
> >>>> there.
> >>>>
> >>>> Jonathan
> >>>>
> >>>> _______________________________________________
> >>>> Swift-user mailing list
> >>>> Swift-user at ci.uchicago.edu
> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Yadu Nand B
> >>>
> >>>
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> >
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140402/7beb52ff/attachment.html>