[Swift-user] Transfer directory structure

Mihael Hategan hategan at mcs.anl.gov
Thu May 29 04:27:56 CDT 2014 

This issue should also now be fixed in trunk.

Mihael

On Wed, 2014-04-02 at 16:52 -0700, Mihael Hategan wrote:
> Ooops. Well, it's meant to work.
> 
> Mihael
> 
> On Wed, 2014-04-02 at 11:05 -0500, David Kelly wrote:
> > I just did a test with swift/0.95-RC5 and filesys mapper on a directory of
> > data I'd like to map, but I'm running into some errors with that. I have
> > the following files I'm trying to bring in:
> > 
> > $ find data
> > data
> > data/foo_a
> > data/foo_a/foo_a.txt
> > data/foo_a/foo_b.txt
> > data/foo_b
> > data/foo_b/bar_1
> > data/foo_b/bar_2
> > data/data.txt
> > 
> > My Swift script:
> > -----------
> > type file;
> > 
> > app check_files (file inputs[])
> > {
> >    ls "data/foo_a/foo_a.txt" "data/foo_a/foo_b.txt" "data/foo_b/bar_1"
> > "data/foo_b/bar_2" "data/data.txt";
> > }
> > 
> > file inputs[] <filesys_mapper; location="data", pattern="**/*.*">;
> > foreach i in inputs {
> >    tracef("%s\n", filename(i));
> > }
> > 
> > check_files(inputs);
> > ----------
> > 
> > Tracef says the filenames are:
> > 
> > data/foo_a.txt
> > data/data.txt
> > data/foo_b.txt
> > 
> > Which seem to omit the directory structure. Then the app fails with:
> > 
> > Execution failed:
> > Exception in ls:
> >     Arguments: [data/foo_a/foo_a.txt, data/foo_a/foo_b.txt,
> > data/foo_b/bar_1, data/foo_b/bar_2, data/data.txt]
> >     Host: westmere
> >     Directory: filesys_extglob-run001/jobs/h/ls-he7o1nol
> > exception @ swift-int.k, line: 530
> > Caused by: null
> > Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> > not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> > parallelFor @ swift-int.k, line: 240
> > Caused by: null
> > Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> > not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> > 
> > In the swift work directory, data/data.txt is the only file I see staged in.
> > 
> > 
> > On Wed, Apr 2, 2014 at 10:40 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> > 
> > > Thanks David (especially for pointing me to the newer user guide).
> > > Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that
> > > accepts the extended glob patterns?
> > >
> > > Jonathan
> > >
> > > On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:
> > >
> > > Hi Jonathan,
> > >
> > > A fairly recent version of 0.95 is available on Midway in the module
> > > "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> > > some warnings about deprecated use of @ in front of functions, and you will
> > > see logs going into a run directory named run001, run002, etc instead of
> > > going to your current working directory. There's more information about
> > > this and the other new (and optional) configuration changes at
> > > http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> > > Please let me know if you have any issues.
> > >
> > > Thanks,
> > > David
> > >
> > >
> > > On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> > >
> > >> Yadu, David, Mihael,
> > >>
> > >> Thanks for your responses.
> > >> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> > >> available on Midway?
> > >>
> > >> Jonathan
> > >>
> > >> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > >>
> > >> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > >> > you should be able to say:
> > >> >
> > >> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> > >> >
> > >> > Mihael
> > >> >
> > >> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> > >> >> I created a ticket about this on Monday because I was running into
> > >> similar
> > >> >> issues in my scripts. An ext mapper worked for me, but I think this is
> > >> a
> > >> >> common pattern we can make easier in future releases.
> > >> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> > >> >>
> > >> >>
> > >> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> > >> wrote:
> > >> >>
> > >> >>> Hi Jonathan,
> > >> >>>
> > >> >>> What I'd do in this case is map every file under the directory you
> > >> want to
> > >> >>> send to the compute nodes, into an array and pass
> > >> >>> that along to your apps. You can do this mapping using either ext
> > >> mapper
> > >> >>> or array mappers.
> > >> >>>
> > >> >>> I have the following dir structure in my folders:
> > >> >>>
> > >> >>> ./dirs/foo_a/foo_a.txt
> > >> >>> ./dirs/foo_a/foo_b.txt
> > >> >>> ./dirs/foo_b/bar_1
> > >> >>> ./dirs/foo_b/bar_2
> > >> >>>
> > >> >>> Here's my ext mapper (mapper.sh) :
> > >> >>> #!/bin/bash
> > >> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> > >> >>>
> > >> >>> If you are using ext mappers, you would need a script which generates
> > >> >>> output in the form [<index>] <filename>
> > >> >>> Here I use find to just output files and awk to get the right format.
> > >> >>>
> > >> >>> The swift mapping would be like this:
> > >> >>> file array[] <ext ; exec="mapper.sh" >;
> > >> >>>
> > >> >>> You could also use array mappers to read all files you need from a
> > >> file
> > >> >>> containing the filenames. I filled filenames.txt with
> > >> >>> the names of all files in the folders.
> > >> >>>
> > >> >>> string[] names = readData("filenames.txt");
> > >> >>> file dirmap[] <array_mapper; files=names>;
> > >> >>>
> > >> >>> I've got both cases as examples tarballed here if you'd like to take a
> > >> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> > >> >>>
> > >> >>> Thanks,
> > >> >>> Yadu
> > >> >>>
> > >> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> > >> wrote:
> > >> >>>
> > >> >>>> Hello all,
> > >> >>>>
> > >> >>>> Is there a simple way to specify "all files including files in
> > >> subfolders
> > >> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> > >> does
> > >> >>>> get everything within a folder, but has problems if there's a folder
> > >> in
> > >> >>>> there.
> > >> >>>>
> > >> >>>> Jonathan
> > >> >>>>
> > >> >>>> _______________________________________________
> > >> >>>> Swift-user mailing list
> > >> >>>> Swift-user at ci.uchicago.edu
> > >> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> --
> > >> >>> Yadu Nand B
> > >> >>>
> > >> >>>
> > >> >>> _______________________________________________
> > >> >>> Swift-user mailing list
> > >> >>> Swift-user at ci.uchicago.edu
> > >> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >> >>>
> > >> >> _______________________________________________
> > >> >> Swift-user mailing list
> > >> >> Swift-user at ci.uchicago.edu
> > >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >> >
> > >> >
> > >>
> > >>
> > >
> > >
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

More information about the Swift-user mailing list