[Swift-user] Transfer directory structure

Mihael Hategan hategan at mcs.anl.gov
Wed Apr 2 18:52:35 CDT 2014 

Ooops. Well, it's meant to work.

Mihael

On Wed, 2014-04-02 at 11:05 -0500, David Kelly wrote:
> I just did a test with swift/0.95-RC5 and filesys mapper on a directory of
> data I'd like to map, but I'm running into some errors with that. I have
> the following files I'm trying to bring in:
> 
> $ find data
> data
> data/foo_a
> data/foo_a/foo_a.txt
> data/foo_a/foo_b.txt
> data/foo_b
> data/foo_b/bar_1
> data/foo_b/bar_2
> data/data.txt
> 
> My Swift script:
> -----------
> type file;
> 
> app check_files (file inputs[])
> {
>    ls "data/foo_a/foo_a.txt" "data/foo_a/foo_b.txt" "data/foo_b/bar_1"
> "data/foo_b/bar_2" "data/data.txt";
> }
> 
> file inputs[] <filesys_mapper; location="data", pattern="**/*.*">;
> foreach i in inputs {
>    tracef("%s\n", filename(i));
> }
> 
> check_files(inputs);
> ----------
> 
> Tracef says the filenames are:
> 
> data/foo_a.txt
> data/data.txt
> data/foo_b.txt
> 
> Which seem to omit the directory structure. Then the app fails with:
> 
> Execution failed:
> Exception in ls:
>     Arguments: [data/foo_a/foo_a.txt, data/foo_a/foo_b.txt,
> data/foo_b/bar_1, data/foo_b/bar_2, data/data.txt]
>     Host: westmere
>     Directory: filesys_extglob-run001/jobs/h/ls-he7o1nol
> exception @ swift-int.k, line: 530
> Caused by: null
> Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> parallelFor @ swift-int.k, line: 240
> Caused by: null
> Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> 
> In the swift work directory, data/data.txt is the only file I see staged in.
> 
> 
> On Wed, Apr 2, 2014 at 10:40 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> 
> > Thanks David (especially for pointing me to the newer user guide).
> > Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that
> > accepts the extended glob patterns?
> >
> > Jonathan
> >
> > On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:
> >
> > Hi Jonathan,
> >
> > A fairly recent version of 0.95 is available on Midway in the module
> > "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> > some warnings about deprecated use of @ in front of functions, and you will
> > see logs going into a run directory named run001, run002, etc instead of
> > going to your current working directory. There's more information about
> > this and the other new (and optional) configuration changes at
> > http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> > Please let me know if you have any issues.
> >
> > Thanks,
> > David
> >
> >
> > On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >
> >> Yadu, David, Mihael,
> >>
> >> Thanks for your responses.
> >> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> >> available on Midway?
> >>
> >> Jonathan
> >>
> >> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>
> >> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> >> > you should be able to say:
> >> >
> >> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> >> >
> >> > Mihael
> >> >
> >> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> >> >> I created a ticket about this on Monday because I was running into
> >> similar
> >> >> issues in my scripts. An ext mapper worked for me, but I think this is
> >> a
> >> >> common pattern we can make easier in future releases.
> >> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> >> >>
> >> >>
> >> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> >> wrote:
> >> >>
> >> >>> Hi Jonathan,
> >> >>>
> >> >>> What I'd do in this case is map every file under the directory you
> >> want to
> >> >>> send to the compute nodes, into an array and pass
> >> >>> that along to your apps. You can do this mapping using either ext
> >> mapper
> >> >>> or array mappers.
> >> >>>
> >> >>> I have the following dir structure in my folders:
> >> >>>
> >> >>> ./dirs/foo_a/foo_a.txt
> >> >>> ./dirs/foo_a/foo_b.txt
> >> >>> ./dirs/foo_b/bar_1
> >> >>> ./dirs/foo_b/bar_2
> >> >>>
> >> >>> Here's my ext mapper (mapper.sh) :
> >> >>> #!/bin/bash
> >> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >> >>>
> >> >>> If you are using ext mappers, you would need a script which generates
> >> >>> output in the form [<index>] <filename>
> >> >>> Here I use find to just output files and awk to get the right format.
> >> >>>
> >> >>> The swift mapping would be like this:
> >> >>> file array[] <ext ; exec="mapper.sh" >;
> >> >>>
> >> >>> You could also use array mappers to read all files you need from a
> >> file
> >> >>> containing the filenames. I filled filenames.txt with
> >> >>> the names of all files in the folders.
> >> >>>
> >> >>> string[] names = readData("filenames.txt");
> >> >>> file dirmap[] <array_mapper; files=names>;
> >> >>>
> >> >>> I've got both cases as examples tarballed here if you'd like to take a
> >> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >> >>>
> >> >>> Thanks,
> >> >>> Yadu
> >> >>>
> >> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> >> wrote:
> >> >>>
> >> >>>> Hello all,
> >> >>>>
> >> >>>> Is there a simple way to specify "all files including files in
> >> subfolders
> >> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> >> does
> >> >>>> get everything within a folder, but has problems if there's a folder
> >> in
> >> >>>> there.
> >> >>>>
> >> >>>> Jonathan
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> Swift-user mailing list
> >> >>>> Swift-user at ci.uchicago.edu
> >> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Yadu Nand B
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> Swift-user mailing list
> >> >>> Swift-user at ci.uchicago.edu
> >> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >>>
> >> >> _______________________________________________
> >> >> Swift-user mailing list
> >> >> Swift-user at ci.uchicago.edu
> >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >
> >> >
> >>
> >>
> >
> >

More information about the Swift-user mailing list