[Swift-user] Transfer directory structure

[Mihael Hategan]

I had this impression that this might also be in 0.94 since it was
pretty isolated change to something that didn't change much between 0.94
and 0.95, but I checked the code and it's not quite so. But it might be
a viable option.

Mihael

On Wed, 2014-04-02 at 10:17 -0500, David Kelly wrote:
> Hi Jonathan,
> 
> A fairly recent version of 0.95 is available on Midway in the module
> "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> some warnings about deprecated use of @ in front of functions, and you will
> see logs going into a run directory named run001, run002, etc instead of
> going to your current working directory. There's more information about
> this and the other new (and optional) configuration changes at
> http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> Please let me know if you have any issues.
> 
> Thanks,
> David
> 
> 
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> 
> > Yadu, David, Mihael,
> >
> > Thanks for your responses.
> > I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> > available on Midway?
> >
> > Jonathan
> >
> > On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >
> > > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > > you should be able to say:
> > >
> > > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> > >
> > > Mihael
> > >
> > > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> > >> I created a ticket about this on Monday because I was running into
> > similar
> > >> issues in my scripts. An ext mapper worked for me, but I think this is a
> > >> common pattern we can make easier in future releases.
> > >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> > >>
> > >>
> > >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> > wrote:
> > >>
> > >>> Hi Jonathan,
> > >>>
> > >>> What I'd do in this case is map every file under the directory you
> > want to
> > >>> send to the compute nodes, into an array and pass
> > >>> that along to your apps. You can do this mapping using either ext
> > mapper
> > >>> or array mappers.
> > >>>
> > >>> I have the following dir structure in my folders:
> > >>>
> > >>> ./dirs/foo_a/foo_a.txt
> > >>> ./dirs/foo_a/foo_b.txt
> > >>> ./dirs/foo_b/bar_1
> > >>> ./dirs/foo_b/bar_2
> > >>>
> > >>> Here's my ext mapper (mapper.sh) :
> > >>> #!/bin/bash
> > >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> > >>>
> > >>> If you are using ext mappers, you would need a script which generates
> > >>> output in the form [<index>] <filename>
> > >>> Here I use find to just output files and awk to get the right format.
> > >>>
> > >>> The swift mapping would be like this:
> > >>> file array[] <ext ; exec="mapper.sh" >;
> > >>>
> > >>> You could also use array mappers to read all files you need from a file
> > >>> containing the filenames. I filled filenames.txt with
> > >>> the names of all files in the folders.
> > >>>
> > >>> string[] names = readData("filenames.txt");
> > >>> file dirmap[] <array_mapper; files=names>;
> > >>>
> > >>> I've got both cases as examples tarballed here if you'd like to take a
> > >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> > >>>
> > >>> Thanks,
> > >>> Yadu
> > >>>
> > >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> > wrote:
> > >>>
> > >>>> Hello all,
> > >>>>
> > >>>> Is there a simple way to specify "all files including files in
> > subfolders
> > >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> > does
> > >>>> get everything within a folder, but has problems if there's a folder
> > in
> > >>>> there.
> > >>>>
> > >>>> Jonathan
> > >>>>
> > >>>> _______________________________________________
> > >>>> Swift-user mailing list
> > >>>> Swift-user at ci.uchicago.edu
> > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Yadu Nand B
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Swift-user mailing list
> > >>> Swift-user at ci.uchicago.edu
> > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >>>
> > >> _______________________________________________
> > >> Swift-user mailing list
> > >> Swift-user at ci.uchicago.edu
> > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >
> > >
> >
> >