[Swift-user] Transfer directory structure

Wed Apr 2 11:05:27 CDT 2014

I just did a test with swift/0.95-RC5 and filesys mapper on a directory of
data I'd like to map, but I'm running into some errors with that. I have
the following files I'm trying to bring in:

$ find data
data
data/foo_a
data/foo_a/foo_a.txt
data/foo_a/foo_b.txt
data/foo_b
data/foo_b/bar_1
data/foo_b/bar_2
data/data.txt

My Swift script:
-----------
type file;

app check_files (file inputs[])
{
   ls "data/foo_a/foo_a.txt" "data/foo_a/foo_b.txt" "data/foo_b/bar_1"
"data/foo_b/bar_2" "data/data.txt";
}

file inputs[] <filesys_mapper; location="data", pattern="**/*.*">;
foreach i in inputs {
   tracef("%s\n", filename(i));
}

check_files(inputs);
----------

Tracef says the filenames are:

data/foo_a.txt
data/data.txt
data/foo_b.txt

Which seem to omit the directory structure. Then the app fails with:

Execution failed:
Exception in ls:
    Arguments: [data/foo_a/foo_a.txt, data/foo_a/foo_b.txt,
data/foo_b/bar_1, data/foo_b/bar_2, data/data.txt]
    Host: westmere
    Directory: filesys_extglob-run001/jobs/h/ls-he7o1nol
exception @ swift-int.k, line: 530
Caused by: null
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
parallelFor @ swift-int.k, line: 240
Caused by: null
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt

In the swift work directory, data/data.txt is the only file I see staged in.

On Wed, Apr 2, 2014 at 10:40 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:

> Thanks David (especially for pointing me to the newer user guide).
> Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that
> accepts the extended glob patterns?
>
> Jonathan
>
> On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:
>
> Hi Jonathan,
>
> A fairly recent version of 0.95 is available on Midway in the module
> "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> some warnings about deprecated use of @ in front of functions, and you will
> see logs going into a run directory named run001, run002, etc instead of
> going to your current working directory. There's more information about
> this and the other new (and optional) configuration changes at
> http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> Please let me know if you have any issues.
>
> Thanks,
> David
>
>
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>
>> Yadu, David, Mihael,
>>
>> Thanks for your responses.
>> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
>> available on Midway?
>>
>> Jonathan
>>
>> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>
>> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
>> > you should be able to say:
>> >
>> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
>> >
>> > Mihael
>> >
>> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
>> >> I created a ticket about this on Monday because I was running into
>> similar
>> >> issues in my scripts. An ext mapper worked for me, but I think this is
>> a
>> >> common pattern we can make easier in future releases.
>> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
>> >>
>> >>
>> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
>> wrote:
>> >>
>> >>> Hi Jonathan,
>> >>>
>> >>> What I'd do in this case is map every file under the directory you
>> want to
>> >>> send to the compute nodes, into an array and pass
>> >>> that along to your apps. You can do this mapping using either ext
>> mapper
>> >>> or array mappers.
>> >>>
>> >>> I have the following dir structure in my folders:
>> >>>
>> >>> ./dirs/foo_a/foo_a.txt
>> >>> ./dirs/foo_a/foo_b.txt
>> >>> ./dirs/foo_b/bar_1
>> >>> ./dirs/foo_b/bar_2
>> >>>
>> >>> Here's my ext mapper (mapper.sh) :
>> >>> #!/bin/bash
>> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>> >>>
>> >>> If you are using ext mappers, you would need a script which generates
>> >>> output in the form [<index>] <filename>
>> >>> Here I use find to just output files and awk to get the right format.
>> >>>
>> >>> The swift mapping would be like this:
>> >>> file array[] <ext ; exec="mapper.sh" >;
>> >>>
>> >>> You could also use array mappers to read all files you need from a
>> file
>> >>> containing the filenames. I filled filenames.txt with
>> >>> the names of all files in the folders.
>> >>>
>> >>> string[] names = readData("filenames.txt");
>> >>> file dirmap[] <array_mapper; files=names>;
>> >>>
>> >>> I've got both cases as examples tarballed here if you'd like to take a
>> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>> >>>
>> >>> Thanks,
>> >>> Yadu
>> >>>
>> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
>> wrote:
>> >>>
>> >>>> Hello all,
>> >>>>
>> >>>> Is there a simple way to specify "all files including files in
>> subfolders
>> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
>> does
>> >>>> get everything within a folder, but has problems if there's a folder
>> in
>> >>>> there.
>> >>>>
>> >>>> Jonathan
>> >>>>
>> >>>> _______________________________________________
>> >>>> Swift-user mailing list
>> >>>> Swift-user at ci.uchicago.edu
>> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Yadu Nand B
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Swift-user mailing list
>> >>> Swift-user at ci.uchicago.edu
>> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >>>
>> >> _______________________________________________
>> >> Swift-user mailing list
>> >> Swift-user at ci.uchicago.edu
>> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >
>> >
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140402/92e18c91/attachment.html>