From jozik at uchicago.edu  Tue Apr  1 08:56:48 2014
From: jozik at uchicago.edu (Jonathan Ozik)
Date: Tue, 1 Apr 2014 08:56:48 -0500
Subject: [Swift-user] Transfer directory structure
Message-ID: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>

Hello all,

Is there a simple way to specify "all files including files in subfolders within a folder" as a file mapper? It looks like the filesys_mapper does get everything within a folder, but has problems if there's a folder in there.

Jonathan



From yadudoc1729 at gmail.com  Tue Apr  1 19:09:59 2014
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Tue, 1 Apr 2014 19:09:59 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
Message-ID: <CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>

Hi Jonathan,

What I'd do in this case is map every file under the directory you want to
send to the compute nodes, into an array and pass
that along to your apps. You can do this mapping using either ext mapper or
array mappers.

I have the following dir structure in my folders:

./dirs/foo_a/foo_a.txt
./dirs/foo_a/foo_b.txt
./dirs/foo_b/bar_1
./dirs/foo_b/bar_2

Here's my ext mapper (mapper.sh) :
#!/bin/bash
find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'

If you are using ext mappers, you would need a script which generates
output in the form [<index>] <filename>
Here I use find to just output files and awk to get the right format.

The swift mapping would be like this:
file array[] <ext ; exec="mapper.sh" >;

You could also use array mappers to read all files you need from a file
containing the filenames. I filled filenames.txt with
the names of all files in the folders.

string[] names = readData("filenames.txt");
file dirmap[] <array_mapper; files=names>;

I've got both cases as examples tarballed here if you'd like to take a look
: http://swift.rcc.uchicago.edu:8042/directory_mapping.tar

Thanks,
Yadu

On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:

> Hello all,
>
> Is there a simple way to specify "all files including files in subfolders
> within a folder" as a file mapper? It looks like the filesys_mapper does
> get everything within a folder, but has problems if there's a folder in
> there.
>
> Jonathan
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>



-- 
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140401/f2ead248/attachment.html>

From davidkelly at uchicago.edu  Tue Apr  1 19:49:00 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Tue, 1 Apr 2014 19:49:00 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
Message-ID: <CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>

I created a ticket about this on Monday because I was running into similar
issues in my scripts. An ext mapper worked for me, but I think this is a
common pattern we can make easier in future releases.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.


On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:

> Hi Jonathan,
>
> What I'd do in this case is map every file under the directory you want to
> send to the compute nodes, into an array and pass
> that along to your apps. You can do this mapping using either ext mapper
> or array mappers.
>
> I have the following dir structure in my folders:
>
> ./dirs/foo_a/foo_a.txt
> ./dirs/foo_a/foo_b.txt
> ./dirs/foo_b/bar_1
> ./dirs/foo_b/bar_2
>
> Here's my ext mapper (mapper.sh) :
> #!/bin/bash
> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>
> If you are using ext mappers, you would need a script which generates
> output in the form [<index>] <filename>
> Here I use find to just output files and awk to get the right format.
>
> The swift mapping would be like this:
> file array[] <ext ; exec="mapper.sh" >;
>
> You could also use array mappers to read all files you need from a file
> containing the filenames. I filled filenames.txt with
> the names of all files in the folders.
>
> string[] names = readData("filenames.txt");
> file dirmap[] <array_mapper; files=names>;
>
> I've got both cases as examples tarballed here if you'd like to take a
> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>
> Thanks,
> Yadu
>
> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>
>> Hello all,
>>
>> Is there a simple way to specify "all files including files in subfolders
>> within a folder" as a file mapper? It looks like the filesys_mapper does
>> get everything within a folder, but has problems if there's a folder in
>> there.
>>
>> Jonathan
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>
>
>
> --
> Yadu Nand B
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140401/8f17f450/attachment.html>

From hategan at mcs.anl.gov  Tue Apr  1 20:50:16 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 1 Apr 2014 18:50:16 -0700
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
Message-ID: <1396403416.19960.2.camel@echo>

At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
you should be able to say:

file[] f <FilesysMapper; location=".", pattern="**/*.*">;

Mihael

On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> I created a ticket about this on Monday because I was running into similar
> issues in my scripts. An ext mapper worked for me, but I think this is a
> common pattern we can make easier in future releases.
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> 
> 
> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
> 
> > Hi Jonathan,
> >
> > What I'd do in this case is map every file under the directory you want to
> > send to the compute nodes, into an array and pass
> > that along to your apps. You can do this mapping using either ext mapper
> > or array mappers.
> >
> > I have the following dir structure in my folders:
> >
> > ./dirs/foo_a/foo_a.txt
> > ./dirs/foo_a/foo_b.txt
> > ./dirs/foo_b/bar_1
> > ./dirs/foo_b/bar_2
> >
> > Here's my ext mapper (mapper.sh) :
> > #!/bin/bash
> > find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >
> > If you are using ext mappers, you would need a script which generates
> > output in the form [<index>] <filename>
> > Here I use find to just output files and awk to get the right format.
> >
> > The swift mapping would be like this:
> > file array[] <ext ; exec="mapper.sh" >;
> >
> > You could also use array mappers to read all files you need from a file
> > containing the filenames. I filled filenames.txt with
> > the names of all files in the folders.
> >
> > string[] names = readData("filenames.txt");
> > file dirmap[] <array_mapper; files=names>;
> >
> > I've got both cases as examples tarballed here if you'd like to take a
> > look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >
> > Thanks,
> > Yadu
> >
> > On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >
> >> Hello all,
> >>
> >> Is there a simple way to specify "all files including files in subfolders
> >> within a folder" as a file mapper? It looks like the filesys_mapper does
> >> get everything within a folder, but has problems if there's a folder in
> >> there.
> >>
> >> Jonathan
> >>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>
> >
> >
> >
> > --
> > Yadu Nand B
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user




From jozik at uchicago.edu  Wed Apr  2 09:51:38 2014
From: jozik at uchicago.edu (Jonathan Ozik)
Date: Wed, 2 Apr 2014 09:51:38 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <1396403416.19960.2.camel@echo>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
Message-ID: <C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>

Yadu, David, Mihael,

Thanks for your responses.
I'm thinking of using the ext mapper for now. Would the trunk/0.95 be available on Midway?

Jonathan

On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> you should be able to say:
> 
> file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> 
> Mihael
> 
> On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
>> I created a ticket about this on Monday because I was running into similar
>> issues in my scripts. An ext mapper worked for me, but I think this is a
>> common pattern we can make easier in future releases.
>> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
>> 
>> 
>> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
>> 
>>> Hi Jonathan,
>>> 
>>> What I'd do in this case is map every file under the directory you want to
>>> send to the compute nodes, into an array and pass
>>> that along to your apps. You can do this mapping using either ext mapper
>>> or array mappers.
>>> 
>>> I have the following dir structure in my folders:
>>> 
>>> ./dirs/foo_a/foo_a.txt
>>> ./dirs/foo_a/foo_b.txt
>>> ./dirs/foo_b/bar_1
>>> ./dirs/foo_b/bar_2
>>> 
>>> Here's my ext mapper (mapper.sh) :
>>> #!/bin/bash
>>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>>> 
>>> If you are using ext mappers, you would need a script which generates
>>> output in the form [<index>] <filename>
>>> Here I use find to just output files and awk to get the right format.
>>> 
>>> The swift mapping would be like this:
>>> file array[] <ext ; exec="mapper.sh" >;
>>> 
>>> You could also use array mappers to read all files you need from a file
>>> containing the filenames. I filled filenames.txt with
>>> the names of all files in the folders.
>>> 
>>> string[] names = readData("filenames.txt");
>>> file dirmap[] <array_mapper; files=names>;
>>> 
>>> I've got both cases as examples tarballed here if you'd like to take a
>>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>>> 
>>> Thanks,
>>> Yadu
>>> 
>>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>>> 
>>>> Hello all,
>>>> 
>>>> Is there a simple way to specify "all files including files in subfolders
>>>> within a folder" as a file mapper? It looks like the filesys_mapper does
>>>> get everything within a folder, but has problems if there's a folder in
>>>> there.
>>>> 
>>>> Jonathan
>>>> 
>>>> _______________________________________________
>>>> Swift-user mailing list
>>>> Swift-user at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Yadu Nand B
>>> 
>>> 
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>> 
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> 



From davidkelly at uchicago.edu  Wed Apr  2 10:17:55 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Wed, 2 Apr 2014 10:17:55 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
	<C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
Message-ID: <CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>

Hi Jonathan,

A fairly recent version of 0.95 is available on Midway in the module
"swift/0.95-RC5". It should backwards compatible with 0.94. You may see
some warnings about deprecated use of @ in front of functions, and you will
see logs going into a run directory named run001, run002, etc instead of
going to your current working directory. There's more information about
this and the other new (and optional) configuration changes at
http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
Please let me know if you have any issues.

Thanks,
David


On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:

> Yadu, David, Mihael,
>
> Thanks for your responses.
> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> available on Midway?
>
> Jonathan
>
> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>
> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > you should be able to say:
> >
> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> >
> > Mihael
> >
> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> >> I created a ticket about this on Monday because I was running into
> similar
> >> issues in my scripts. An ext mapper worked for me, but I think this is a
> >> common pattern we can make easier in future releases.
> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> >>
> >>
> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> wrote:
> >>
> >>> Hi Jonathan,
> >>>
> >>> What I'd do in this case is map every file under the directory you
> want to
> >>> send to the compute nodes, into an array and pass
> >>> that along to your apps. You can do this mapping using either ext
> mapper
> >>> or array mappers.
> >>>
> >>> I have the following dir structure in my folders:
> >>>
> >>> ./dirs/foo_a/foo_a.txt
> >>> ./dirs/foo_a/foo_b.txt
> >>> ./dirs/foo_b/bar_1
> >>> ./dirs/foo_b/bar_2
> >>>
> >>> Here's my ext mapper (mapper.sh) :
> >>> #!/bin/bash
> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >>>
> >>> If you are using ext mappers, you would need a script which generates
> >>> output in the form [<index>] <filename>
> >>> Here I use find to just output files and awk to get the right format.
> >>>
> >>> The swift mapping would be like this:
> >>> file array[] <ext ; exec="mapper.sh" >;
> >>>
> >>> You could also use array mappers to read all files you need from a file
> >>> containing the filenames. I filled filenames.txt with
> >>> the names of all files in the folders.
> >>>
> >>> string[] names = readData("filenames.txt");
> >>> file dirmap[] <array_mapper; files=names>;
> >>>
> >>> I've got both cases as examples tarballed here if you'd like to take a
> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >>>
> >>> Thanks,
> >>> Yadu
> >>>
> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> wrote:
> >>>
> >>>> Hello all,
> >>>>
> >>>> Is there a simple way to specify "all files including files in
> subfolders
> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> does
> >>>> get everything within a folder, but has problems if there's a folder
> in
> >>>> there.
> >>>>
> >>>> Jonathan
> >>>>
> >>>> _______________________________________________
> >>>> Swift-user mailing list
> >>>> Swift-user at ci.uchicago.edu
> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Yadu Nand B
> >>>
> >>>
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140402/8d63e206/attachment.html>

From jozik at uchicago.edu  Wed Apr  2 10:40:06 2014
From: jozik at uchicago.edu (Jonathan Ozik)
Date: Wed, 2 Apr 2014 10:40:06 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
	<C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
	<CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
Message-ID: <94AB5E8C-6EB0-48C9-B645-181D2D89CAA1@uchicago.edu>

Thanks David (especially for pointing me to the newer user guide).
Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that accepts the extended glob patterns?

Jonathan

On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:

> Hi Jonathan,
> 
> A fairly recent version of 0.95 is available on Midway in the module "swift/0.95-RC5". It should backwards compatible with 0.94. You may see some warnings about deprecated use of @ in front of functions, and you will see logs going into a run directory named run001, run002, etc instead of going to your current working directory. There's more information about this and the other new (and optional) configuration changes at http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration. Please let me know if you have any issues.
> 
> Thanks,
> David
> 
> 
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> Yadu, David, Mihael,
> 
> Thanks for your responses.
> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be available on Midway?
> 
> Jonathan
> 
> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > you should be able to say:
> >
> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> >
> > Mihael
> >
> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> >> I created a ticket about this on Monday because I was running into similar
> >> issues in my scripts. An ext mapper worked for me, but I think this is a
> >> common pattern we can make easier in future releases.
> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> >>
> >>
> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com> wrote:
> >>
> >>> Hi Jonathan,
> >>>
> >>> What I'd do in this case is map every file under the directory you want to
> >>> send to the compute nodes, into an array and pass
> >>> that along to your apps. You can do this mapping using either ext mapper
> >>> or array mappers.
> >>>
> >>> I have the following dir structure in my folders:
> >>>
> >>> ./dirs/foo_a/foo_a.txt
> >>> ./dirs/foo_a/foo_b.txt
> >>> ./dirs/foo_b/bar_1
> >>> ./dirs/foo_b/bar_2
> >>>
> >>> Here's my ext mapper (mapper.sh) :
> >>> #!/bin/bash
> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >>>
> >>> If you are using ext mappers, you would need a script which generates
> >>> output in the form [<index>] <filename>
> >>> Here I use find to just output files and awk to get the right format.
> >>>
> >>> The swift mapping would be like this:
> >>> file array[] <ext ; exec="mapper.sh" >;
> >>>
> >>> You could also use array mappers to read all files you need from a file
> >>> containing the filenames. I filled filenames.txt with
> >>> the names of all files in the folders.
> >>>
> >>> string[] names = readData("filenames.txt");
> >>> file dirmap[] <array_mapper; files=names>;
> >>>
> >>> I've got both cases as examples tarballed here if you'd like to take a
> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >>>
> >>> Thanks,
> >>> Yadu
> >>>
> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >>>
> >>>> Hello all,
> >>>>
> >>>> Is there a simple way to specify "all files including files in subfolders
> >>>> within a folder" as a file mapper? It looks like the filesys_mapper does
> >>>> get everything within a folder, but has problems if there's a folder in
> >>>> there.
> >>>>
> >>>> Jonathan
> >>>>
> >>>> _______________________________________________
> >>>> Swift-user mailing list
> >>>> Swift-user at ci.uchicago.edu
> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Yadu Nand B
> >>>
> >>>
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >>>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
> >
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140402/7beb52ff/attachment.html>

From davidkelly at uchicago.edu  Wed Apr  2 11:05:27 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Wed, 2 Apr 2014 11:05:27 -0500
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <94AB5E8C-6EB0-48C9-B645-181D2D89CAA1@uchicago.edu>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
	<C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
	<CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
	<94AB5E8C-6EB0-48C9-B645-181D2D89CAA1@uchicago.edu>
Message-ID: <CA+_+Ey9wsNXY5GxG+RDr14+ZqRBvsN+Vk4O3jv68bZN5_EBWuw@mail.gmail.com>

I just did a test with swift/0.95-RC5 and filesys mapper on a directory of
data I'd like to map, but I'm running into some errors with that. I have
the following files I'm trying to bring in:

$ find data
data
data/foo_a
data/foo_a/foo_a.txt
data/foo_a/foo_b.txt
data/foo_b
data/foo_b/bar_1
data/foo_b/bar_2
data/data.txt

My Swift script:
-----------
type file;

app check_files (file inputs[])
{
   ls "data/foo_a/foo_a.txt" "data/foo_a/foo_b.txt" "data/foo_b/bar_1"
"data/foo_b/bar_2" "data/data.txt";
}

file inputs[] <filesys_mapper; location="data", pattern="**/*.*">;
foreach i in inputs {
   tracef("%s\n", filename(i));
}

check_files(inputs);
----------

Tracef says the filenames are:

data/foo_a.txt
data/data.txt
data/foo_b.txt

Which seem to omit the directory structure. Then the app fails with:

Execution failed:
Exception in ls:
    Arguments: [data/foo_a/foo_a.txt, data/foo_a/foo_b.txt,
data/foo_b/bar_1, data/foo_b/bar_2, data/data.txt]
    Host: westmere
    Directory: filesys_extglob-run001/jobs/h/ls-he7o1nol
exception @ swift-int.k, line: 530
Caused by: null
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
parallelFor @ swift-int.k, line: 240
Caused by: null
Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt

In the swift work directory, data/data.txt is the only file I see staged in.


On Wed, Apr 2, 2014 at 10:40 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:

> Thanks David (especially for pointing me to the newer user guide).
> Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that
> accepts the extended glob patterns?
>
> Jonathan
>
> On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:
>
> Hi Jonathan,
>
> A fairly recent version of 0.95 is available on Midway in the module
> "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> some warnings about deprecated use of @ in front of functions, and you will
> see logs going into a run directory named run001, run002, etc instead of
> going to your current working directory. There's more information about
> this and the other new (and optional) configuration changes at
> http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> Please let me know if you have any issues.
>
> Thanks,
> David
>
>
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
>
>> Yadu, David, Mihael,
>>
>> Thanks for your responses.
>> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
>> available on Midway?
>>
>> Jonathan
>>
>> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>
>> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
>> > you should be able to say:
>> >
>> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
>> >
>> > Mihael
>> >
>> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
>> >> I created a ticket about this on Monday because I was running into
>> similar
>> >> issues in my scripts. An ext mapper worked for me, but I think this is
>> a
>> >> common pattern we can make easier in future releases.
>> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
>> >>
>> >>
>> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
>> wrote:
>> >>
>> >>> Hi Jonathan,
>> >>>
>> >>> What I'd do in this case is map every file under the directory you
>> want to
>> >>> send to the compute nodes, into an array and pass
>> >>> that along to your apps. You can do this mapping using either ext
>> mapper
>> >>> or array mappers.
>> >>>
>> >>> I have the following dir structure in my folders:
>> >>>
>> >>> ./dirs/foo_a/foo_a.txt
>> >>> ./dirs/foo_a/foo_b.txt
>> >>> ./dirs/foo_b/bar_1
>> >>> ./dirs/foo_b/bar_2
>> >>>
>> >>> Here's my ext mapper (mapper.sh) :
>> >>> #!/bin/bash
>> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
>> >>>
>> >>> If you are using ext mappers, you would need a script which generates
>> >>> output in the form [<index>] <filename>
>> >>> Here I use find to just output files and awk to get the right format.
>> >>>
>> >>> The swift mapping would be like this:
>> >>> file array[] <ext ; exec="mapper.sh" >;
>> >>>
>> >>> You could also use array mappers to read all files you need from a
>> file
>> >>> containing the filenames. I filled filenames.txt with
>> >>> the names of all files in the folders.
>> >>>
>> >>> string[] names = readData("filenames.txt");
>> >>> file dirmap[] <array_mapper; files=names>;
>> >>>
>> >>> I've got both cases as examples tarballed here if you'd like to take a
>> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
>> >>>
>> >>> Thanks,
>> >>> Yadu
>> >>>
>> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
>> wrote:
>> >>>
>> >>>> Hello all,
>> >>>>
>> >>>> Is there a simple way to specify "all files including files in
>> subfolders
>> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
>> does
>> >>>> get everything within a folder, but has problems if there's a folder
>> in
>> >>>> there.
>> >>>>
>> >>>> Jonathan
>> >>>>
>> >>>> _______________________________________________
>> >>>> Swift-user mailing list
>> >>>> Swift-user at ci.uchicago.edu
>> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Yadu Nand B
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Swift-user mailing list
>> >>> Swift-user at ci.uchicago.edu
>> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >>>
>> >> _______________________________________________
>> >> Swift-user mailing list
>> >> Swift-user at ci.uchicago.edu
>> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> >
>> >
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140402/92e18c91/attachment.html>

From hategan at mcs.anl.gov  Wed Apr  2 11:47:43 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 2 Apr 2014 09:47:43 -0700
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
	<C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
	<CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
Message-ID: <1396457263.23317.1.camel@echo>

I had this impression that this might also be in 0.94 since it was
pretty isolated change to something that didn't change much between 0.94
and 0.95, but I checked the code and it's not quite so. But it might be
a viable option.

Mihael

On Wed, 2014-04-02 at 10:17 -0500, David Kelly wrote:
> Hi Jonathan,
> 
> A fairly recent version of 0.95 is available on Midway in the module
> "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> some warnings about deprecated use of @ in front of functions, and you will
> see logs going into a run directory named run001, run002, etc instead of
> going to your current working directory. There's more information about
> this and the other new (and optional) configuration changes at
> http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> Please let me know if you have any issues.
> 
> Thanks,
> David
> 
> 
> On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> 
> > Yadu, David, Mihael,
> >
> > Thanks for your responses.
> > I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> > available on Midway?
> >
> > Jonathan
> >
> > On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >
> > > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> > > you should be able to say:
> > >
> > > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> > >
> > > Mihael
> > >
> > > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> > >> I created a ticket about this on Monday because I was running into
> > similar
> > >> issues in my scripts. An ext mapper worked for me, but I think this is a
> > >> common pattern we can make easier in future releases.
> > >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> > >>
> > >>
> > >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> > wrote:
> > >>
> > >>> Hi Jonathan,
> > >>>
> > >>> What I'd do in this case is map every file under the directory you
> > want to
> > >>> send to the compute nodes, into an array and pass
> > >>> that along to your apps. You can do this mapping using either ext
> > mapper
> > >>> or array mappers.
> > >>>
> > >>> I have the following dir structure in my folders:
> > >>>
> > >>> ./dirs/foo_a/foo_a.txt
> > >>> ./dirs/foo_a/foo_b.txt
> > >>> ./dirs/foo_b/bar_1
> > >>> ./dirs/foo_b/bar_2
> > >>>
> > >>> Here's my ext mapper (mapper.sh) :
> > >>> #!/bin/bash
> > >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> > >>>
> > >>> If you are using ext mappers, you would need a script which generates
> > >>> output in the form [<index>] <filename>
> > >>> Here I use find to just output files and awk to get the right format.
> > >>>
> > >>> The swift mapping would be like this:
> > >>> file array[] <ext ; exec="mapper.sh" >;
> > >>>
> > >>> You could also use array mappers to read all files you need from a file
> > >>> containing the filenames. I filled filenames.txt with
> > >>> the names of all files in the folders.
> > >>>
> > >>> string[] names = readData("filenames.txt");
> > >>> file dirmap[] <array_mapper; files=names>;
> > >>>
> > >>> I've got both cases as examples tarballed here if you'd like to take a
> > >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> > >>>
> > >>> Thanks,
> > >>> Yadu
> > >>>
> > >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> > wrote:
> > >>>
> > >>>> Hello all,
> > >>>>
> > >>>> Is there a simple way to specify "all files including files in
> > subfolders
> > >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> > does
> > >>>> get everything within a folder, but has problems if there's a folder
> > in
> > >>>> there.
> > >>>>
> > >>>> Jonathan
> > >>>>
> > >>>> _______________________________________________
> > >>>> Swift-user mailing list
> > >>>> Swift-user at ci.uchicago.edu
> > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Yadu Nand B
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Swift-user mailing list
> > >>> Swift-user at ci.uchicago.edu
> > >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >>>
> > >> _______________________________________________
> > >> Swift-user mailing list
> > >> Swift-user at ci.uchicago.edu
> > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > >
> > >
> >
> >




From baiypwup at hotmail.com  Wed Apr  2 12:28:39 2014
From: baiypwup at hotmail.com (chen sui)
Date: Thu, 3 Apr 2014 01:28:39 +0800
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
Message-ID: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>

Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.

I think I have narrowed down the cause of the problem;

When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
[begin example]
#PBS -S /bin/bash
#PBS -N B0402-0012360-0
#PBS -m n
(omitted)
export WORKER_LOGGING_LEVEL=NONE   (*)
#PBS -v WORKER_LOGGING_LEVEL  (*)
[end example]

The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.

[begin Console output]
[cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
506530.eric2
** Job deleted, insufficient hours (0 < 96) left in allocation null **
[end Console output]

Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.

The reason seems to be 
   1) the cluster I'm using does not support the "-v" switch.
   2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).

That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.

So my question is, is it possible to do this through Swift configurations?
I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".

Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?

Thanks!Tommy Chen
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140403/5468201c/attachment.html>

From baiypwup at hotmail.com  Wed Apr  2 17:57:00 2014
From: baiypwup at hotmail.com (chen sui)
Date: Thu, 3 Apr 2014 06:57:00 +0800
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
In-Reply-To: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>
References: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>
Message-ID: <SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>

Hello SWIFT,
I found a solution to the problem. It seems to be working for me in this specific situation so far.
The solution is to comment out lines 242 through 253 in file "cog/modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java" and recompile the JAR file "cog-provider-localscheduler-0.4.jar".
After this change, PBSExecutor would no longer insert the starred lines and it now works on the cluster I'm using
Hope this may be useful for people who encounter the same issue.
Thanks!Tommy Chen

From: baiypwup at hotmail.com
To: swift-user at ci.uchicago.edu
Subject: SWIFT Generates "incompatible" PBS scripts
Date: Thu, 3 Apr 2014 01:28:39 +0800




Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.

I think I have narrowed down the cause of the problem;

When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
[begin example]
#PBS -S /bin/bash
#PBS -N B0402-0012360-0
#PBS -m n
(omitted)
export WORKER_LOGGING_LEVEL=NONE   (*)
#PBS -v WORKER_LOGGING_LEVEL  (*)
[end example]

The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.

[begin Console output]
[cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
506530.eric2
** Job deleted, insufficient hours (0 < 96) left in allocation null **
[end Console output]

Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.

The reason seems to be 
   1) the cluster I'm using does not support the "-v" switch.
   2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).

That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.

So my question is, is it possible to do this through Swift configurations?
I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".

Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?

Thanks!Tommy Chen
 		 	   		   		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140403/135f374e/attachment.html>

From hategan at mcs.anl.gov  Wed Apr  2 18:21:06 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 2 Apr 2014 16:21:06 -0700
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
In-Reply-To: <SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>
References: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>
	<SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>
Message-ID: <1396480866.13706.3.camel@echo>

Hi,

Yes, I suspect that the problem is that the script, as written
originally by PBSExecutor is not sh-safe, but relies on some specific
version of bash. Out of curiosity, on the machine this doesn't work on,
can you let us know what the versions of /bin/sh and /bin/bash.

Your workaround seems ok, in that I don't think it would cause problems
unless you want to use the PBS provider directly and pass environment
variables to jobs. In any event, we should fix this.

Mihael

On Thu, 2014-04-03 at 06:57 +0800, chen sui wrote:
> Hello SWIFT,
> I found a solution to the problem. It seems to be working for me in this specific situation so far.
> The solution is to comment out lines 242 through 253 in file "cog/modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java" and recompile the JAR file "cog-provider-localscheduler-0.4.jar".
> After this change, PBSExecutor would no longer insert the starred lines and it now works on the cluster I'm using
> Hope this may be useful for people who encounter the same issue.
> Thanks!Tommy Chen
> 
> From: baiypwup at hotmail.com
> To: swift-user at ci.uchicago.edu
> Subject: SWIFT Generates "incompatible" PBS scripts
> Date: Thu, 3 Apr 2014 01:28:39 +0800
> 
> 
> 
> 
> Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.
> 
> I think I have narrowed down the cause of the problem;
> 
> When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
> [begin example]
> #PBS -S /bin/bash
> #PBS -N B0402-0012360-0
> #PBS -m n
> (omitted)
> export WORKER_LOGGING_LEVEL=NONE   (*)
> #PBS -v WORKER_LOGGING_LEVEL  (*)
> [end example]
> 
> The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.
> 
> [begin Console output]
> [cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
> 506530.eric2
> ** Job deleted, insufficient hours (0 < 96) left in allocation null **
> [end Console output]
> 
> Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.
> 
> The reason seems to be 
>    1) the cluster I'm using does not support the "-v" switch.
>    2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).
> 
> That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.
> 
> So my question is, is it possible to do this through Swift configurations?
> I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".
> 
> Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?
> 
> Thanks!Tommy Chen
>  		 	   		   		 	   		  
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user




From wilde at anl.gov  Wed Apr  2 18:25:17 2014
From: wilde at anl.gov (Michael Wilde)
Date: Wed, 2 Apr 2014 18:25:17 -0500
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
In-Reply-To: <1396480866.13706.3.camel@echo>
References: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>	<SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>
	<1396480866.13706.3.camel@echo>
Message-ID: <533C9C5D.30208@anl.gov>

I've also run into the issue of PBS ignoring all directives after the 
first non-directive line in the file. I think that we either already 
have a ticket on this or need one; its indeed a Swift bug as far as I 
can tell.

Thanks for reporting it; as Mihael says, we'll create a fix and report 
back to the swift-user list.

- Mike

On 4/2/14, 6:21 PM, Mihael Hategan wrote:
> Hi,
>
> Yes, I suspect that the problem is that the script, as written
> originally by PBSExecutor is not sh-safe, but relies on some specific
> version of bash. Out of curiosity, on the machine this doesn't work on,
> can you let us know what the versions of /bin/sh and /bin/bash.
>
> Your workaround seems ok, in that I don't think it would cause problems
> unless you want to use the PBS provider directly and pass environment
> variables to jobs. In any event, we should fix this.
>
> Mihael
>
> On Thu, 2014-04-03 at 06:57 +0800, chen sui wrote:
>> Hello SWIFT,
>> I found a solution to the problem. It seems to be working for me in this specific situation so far.
>> The solution is to comment out lines 242 through 253 in file "cog/modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java" and recompile the JAR file "cog-provider-localscheduler-0.4.jar".
>> After this change, PBSExecutor would no longer insert the starred lines and it now works on the cluster I'm using
>> Hope this may be useful for people who encounter the same issue.
>> Thanks!Tommy Chen
>>
>> From: baiypwup at hotmail.com
>> To: swift-user at ci.uchicago.edu
>> Subject: SWIFT Generates "incompatible" PBS scripts
>> Date: Thu, 3 Apr 2014 01:28:39 +0800
>>
>>
>>
>>
>> Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.
>>
>> I think I have narrowed down the cause of the problem;
>>
>> When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
>> [begin example]
>> #PBS -S /bin/bash
>> #PBS -N B0402-0012360-0
>> #PBS -m n
>> (omitted)
>> export WORKER_LOGGING_LEVEL=NONE   (*)
>> #PBS -v WORKER_LOGGING_LEVEL  (*)
>> [end example]
>>
>> The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.
>>
>> [begin Console output]
>> [cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
>> 506530.eric2
>> ** Job deleted, insufficient hours (0 < 96) left in allocation null **
>> [end Console output]
>>
>> Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.
>>
>> The reason seems to be
>>     1) the cluster I'm using does not support the "-v" switch.
>>     2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).
>>
>> That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.
>>
>> So my question is, is it possible to do this through Swift configurations?
>> I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".
>>
>> Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?
>>
>> Thanks!Tommy Chen
>>   		 	   		   		 	   		
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From wilde at anl.gov  Wed Apr  2 18:26:44 2014
From: wilde at anl.gov (Michael Wilde)
Date: Wed, 2 Apr 2014 18:26:44 -0500
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
In-Reply-To: <533C9C5D.30208@anl.gov>
References: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>	<SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>	<1396480866.13706.3.camel@echo>
	<533C9C5D.30208@anl.gov>
Message-ID: <533C9CB4.8080105@anl.gov>

Here's the ticket I mentioned:

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1044

- Mike

On 4/2/14, 6:25 PM, Michael Wilde wrote:
> I've also run into the issue of PBS ignoring all directives after the
> first non-directive line in the file. I think that we either already
> have a ticket on this or need one; its indeed a Swift bug as far as I
> can tell.
>
> Thanks for reporting it; as Mihael says, we'll create a fix and report
> back to the swift-user list.
>
> - Mike
>
> On 4/2/14, 6:21 PM, Mihael Hategan wrote:
>> Hi,
>>
>> Yes, I suspect that the problem is that the script, as written
>> originally by PBSExecutor is not sh-safe, but relies on some specific
>> version of bash. Out of curiosity, on the machine this doesn't work on,
>> can you let us know what the versions of /bin/sh and /bin/bash.
>>
>> Your workaround seems ok, in that I don't think it would cause problems
>> unless you want to use the PBS provider directly and pass environment
>> variables to jobs. In any event, we should fix this.
>>
>> Mihael
>>
>> On Thu, 2014-04-03 at 06:57 +0800, chen sui wrote:
>>> Hello SWIFT,
>>> I found a solution to the problem. It seems to be working for me in this specific situation so far.
>>> The solution is to comment out lines 242 through 253 in file "cog/modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java" and recompile the JAR file "cog-provider-localscheduler-0.4.jar".
>>> After this change, PBSExecutor would no longer insert the starred lines and it now works on the cluster I'm using
>>> Hope this may be useful for people who encounter the same issue.
>>> Thanks!Tommy Chen
>>>
>>> From: baiypwup at hotmail.com
>>> To: swift-user at ci.uchicago.edu
>>> Subject: SWIFT Generates "incompatible" PBS scripts
>>> Date: Thu, 3 Apr 2014 01:28:39 +0800
>>>
>>>
>>>
>>>
>>> Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.
>>>
>>> I think I have narrowed down the cause of the problem;
>>>
>>> When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
>>> [begin example]
>>> #PBS -S /bin/bash
>>> #PBS -N B0402-0012360-0
>>> #PBS -m n
>>> (omitted)
>>> export WORKER_LOGGING_LEVEL=NONE   (*)
>>> #PBS -v WORKER_LOGGING_LEVEL  (*)
>>> [end example]
>>>
>>> The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.
>>>
>>> [begin Console output]
>>> [cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
>>> 506530.eric2
>>> ** Job deleted, insufficient hours (0 < 96) left in allocation null **
>>> [end Console output]
>>>
>>> Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.
>>>
>>> The reason seems to be
>>>      1) the cluster I'm using does not support the "-v" switch.
>>>      2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).
>>>
>>> That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.
>>>
>>> So my question is, is it possible to do this through Swift configurations?
>>> I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".
>>>
>>> Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?
>>>
>>> Thanks!Tommy Chen
>>>    		 	   		   		 	   		
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Wed Apr  2 18:52:35 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 2 Apr 2014 16:52:35 -0700
Subject: [Swift-user] Transfer directory structure
In-Reply-To: <CA+_+Ey9wsNXY5GxG+RDr14+ZqRBvsN+Vk4O3jv68bZN5_EBWuw@mail.gmail.com>
References: <1F6C5B44-8853-4D3B-B152-140C7FF31F92@uchicago.edu>
	<CANa904mDD+uZ0R0Ff8jANP+Y+=65G4z-jAv712_BhthEVLrY3Q@mail.gmail.com>
	<CA+_+Ey8ntzo4ChQHx443i42GhMqC-+BzxiwkBZ_g=8QhAF_cJQ@mail.gmail.com>
	<1396403416.19960.2.camel@echo>
	<C6031CF1-153A-47B6-A91A-F54FD534BAB8@uchicago.edu>
	<CA+_+Ey-ogBVhAYM2Cj=b87ioj5L=b6J5+nvb5Pu3KgcKXGPhZQ@mail.gmail.com>
	<94AB5E8C-6EB0-48C9-B645-181D2D89CAA1@uchicago.edu>
	<CA+_+Ey9wsNXY5GxG+RDr14+ZqRBvsN+Vk4O3jv68bZN5_EBWuw@mail.gmail.com>
Message-ID: <1396482755.13706.5.camel@echo>

Ooops. Well, it's meant to work.

Mihael

On Wed, 2014-04-02 at 11:05 -0500, David Kelly wrote:
> I just did a test with swift/0.95-RC5 and filesys mapper on a directory of
> data I'd like to map, but I'm running into some errors with that. I have
> the following files I'm trying to bring in:
> 
> $ find data
> data
> data/foo_a
> data/foo_a/foo_a.txt
> data/foo_a/foo_b.txt
> data/foo_b
> data/foo_b/bar_1
> data/foo_b/bar_2
> data/data.txt
> 
> My Swift script:
> -----------
> type file;
> 
> app check_files (file inputs[])
> {
>    ls "data/foo_a/foo_a.txt" "data/foo_a/foo_b.txt" "data/foo_b/bar_1"
> "data/foo_b/bar_2" "data/data.txt";
> }
> 
> file inputs[] <filesys_mapper; location="data", pattern="**/*.*">;
> foreach i in inputs {
>    tracef("%s\n", filename(i));
> }
> 
> check_files(inputs);
> ----------
> 
> Tracef says the filenames are:
> 
> data/foo_a.txt
> data/data.txt
> data/foo_b.txt
> 
> Which seem to omit the directory structure. Then the app fails with:
> 
> Execution failed:
> Exception in ls:
>     Arguments: [data/foo_a/foo_a.txt, data/foo_a/foo_b.txt,
> data/foo_b/bar_1, data/foo_b/bar_2, data/data.txt]
>     Host: westmere
>     Directory: filesys_extglob-run001/jobs/h/ls-he7o1nol
> exception @ swift-int.k, line: 530
> Caused by: null
> Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> parallelFor @ swift-int.k, line: 240
> Caused by: null
> Caused by: org.globus.cog.abstraction.impl.file.FileNotFoundException: File
> not found: /home/davidkelly999/tests/filesys_extglob/./data/foo_a.txt
> 
> In the swift work directory, data/data.txt is the only file I see staged in.
> 
> 
> On Wed, Apr 2, 2014 at 10:40 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> 
> > Thanks David (especially for pointing me to the newer user guide).
> > Mihael, would the "swift/0.95-RC5" version contain the FilesysMapper that
> > accepts the extended glob patterns?
> >
> > Jonathan
> >
> > On Apr 2, 2014, at 10:17 AM, David Kelly <davidkelly at uchicago.edu> wrote:
> >
> > Hi Jonathan,
> >
> > A fairly recent version of 0.95 is available on Midway in the module
> > "swift/0.95-RC5". It should backwards compatible with 0.94. You may see
> > some warnings about deprecated use of @ in front of functions, and you will
> > see logs going into a run directory named run001, run002, etc instead of
> > going to your current working directory. There's more information about
> > this and the other new (and optional) configuration changes at
> > http://swiftlang.org/guides/trunk/userguide/userguide.html#_configuration.
> > Please let me know if you have any issues.
> >
> > Thanks,
> > David
> >
> >
> > On Wed, Apr 2, 2014 at 9:51 AM, Jonathan Ozik <jozik at uchicago.edu> wrote:
> >
> >> Yadu, David, Mihael,
> >>
> >> Thanks for your responses.
> >> I'm thinking of using the ext mapper for now. Would the trunk/0.95 be
> >> available on Midway?
> >>
> >> Jonathan
> >>
> >> On Apr 1, 2014, at 8:50 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>
> >> > At least in trunk/0.95, FilesysMapper accepts extended glob patterns, so
> >> > you should be able to say:
> >> >
> >> > file[] f <FilesysMapper; location=".", pattern="**/*.*">;
> >> >
> >> > Mihael
> >> >
> >> > On Tue, 2014-04-01 at 19:49 -0500, David Kelly wrote:
> >> >> I created a ticket about this on Monday because I was running into
> >> similar
> >> >> issues in my scripts. An ext mapper worked for me, but I think this is
> >> a
> >> >> common pattern we can make easier in future releases.
> >> >> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229.
> >> >>
> >> >>
> >> >> On Tue, Apr 1, 2014 at 7:09 PM, Yadu Nand <yadudoc1729 at gmail.com>
> >> wrote:
> >> >>
> >> >>> Hi Jonathan,
> >> >>>
> >> >>> What I'd do in this case is map every file under the directory you
> >> want to
> >> >>> send to the compute nodes, into an array and pass
> >> >>> that along to your apps. You can do this mapping using either ext
> >> mapper
> >> >>> or array mappers.
> >> >>>
> >> >>> I have the following dir structure in my folders:
> >> >>>
> >> >>> ./dirs/foo_a/foo_a.txt
> >> >>> ./dirs/foo_a/foo_b.txt
> >> >>> ./dirs/foo_b/bar_1
> >> >>> ./dirs/foo_b/bar_2
> >> >>>
> >> >>> Here's my ext mapper (mapper.sh) :
> >> >>> #!/bin/bash
> >> >>> find ./dirs -type f | awk '{printf("[%d] %s\n", NR, $0)}'
> >> >>>
> >> >>> If you are using ext mappers, you would need a script which generates
> >> >>> output in the form [<index>] <filename>
> >> >>> Here I use find to just output files and awk to get the right format.
> >> >>>
> >> >>> The swift mapping would be like this:
> >> >>> file array[] <ext ; exec="mapper.sh" >;
> >> >>>
> >> >>> You could also use array mappers to read all files you need from a
> >> file
> >> >>> containing the filenames. I filled filenames.txt with
> >> >>> the names of all files in the folders.
> >> >>>
> >> >>> string[] names = readData("filenames.txt");
> >> >>> file dirmap[] <array_mapper; files=names>;
> >> >>>
> >> >>> I've got both cases as examples tarballed here if you'd like to take a
> >> >>> look : http://swift.rcc.uchicago.edu:8042/directory_mapping.tar
> >> >>>
> >> >>> Thanks,
> >> >>> Yadu
> >> >>>
> >> >>> On Tue, Apr 1, 2014 at 8:56 AM, Jonathan Ozik <jozik at uchicago.edu>
> >> wrote:
> >> >>>
> >> >>>> Hello all,
> >> >>>>
> >> >>>> Is there a simple way to specify "all files including files in
> >> subfolders
> >> >>>> within a folder" as a file mapper? It looks like the filesys_mapper
> >> does
> >> >>>> get everything within a folder, but has problems if there's a folder
> >> in
> >> >>>> there.
> >> >>>>
> >> >>>> Jonathan
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> Swift-user mailing list
> >> >>>> Swift-user at ci.uchicago.edu
> >> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Yadu Nand B
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> Swift-user mailing list
> >> >>> Swift-user at ci.uchicago.edu
> >> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >>>
> >> >> _______________________________________________
> >> >> Swift-user mailing list
> >> >> Swift-user at ci.uchicago.edu
> >> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> >
> >> >
> >>
> >>
> >
> >




From hategan at mcs.anl.gov  Wed Apr  2 19:20:21 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 2 Apr 2014 17:20:21 -0700
Subject: [Swift-user] SWIFT Generates "incompatible" PBS scripts
In-Reply-To: <533C9CB4.8080105@anl.gov>
References: <SNT148-W12FD3401283DF36F2D733EDE6D0@phx.gbl>
	<SNT148-W83D7494D48C597F10EF00BDE6D0@phx.gbl>
	<1396480866.13706.3.camel@echo> <533C9C5D.30208@anl.gov>
	<533C9CB4.8080105@anl.gov>
Message-ID: <1396484421.14795.2.camel@echo>

I seem to recall that the current solution is the result of "#PBS -v
name=value" having some problems. Does anybody else remember something
of that sort?

Mihael

On Wed, 2014-04-02 at 18:26 -0500, Michael Wilde wrote:
> Here's the ticket I mentioned:
> 
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1044
> 
> - Mike
> 
> On 4/2/14, 6:25 PM, Michael Wilde wrote:
> > I've also run into the issue of PBS ignoring all directives after the
> > first non-directive line in the file. I think that we either already
> > have a ticket on this or need one; its indeed a Swift bug as far as I
> > can tell.
> >
> > Thanks for reporting it; as Mihael says, we'll create a fix and report
> > back to the swift-user list.
> >
> > - Mike
> >
> > On 4/2/14, 6:21 PM, Mihael Hategan wrote:
> >> Hi,
> >>
> >> Yes, I suspect that the problem is that the script, as written
> >> originally by PBSExecutor is not sh-safe, but relies on some specific
> >> version of bash. Out of curiosity, on the machine this doesn't work on,
> >> can you let us know what the versions of /bin/sh and /bin/bash.
> >>
> >> Your workaround seems ok, in that I don't think it would cause problems
> >> unless you want to use the PBS provider directly and pass environment
> >> variables to jobs. In any event, we should fix this.
> >>
> >> Mihael
> >>
> >> On Thu, 2014-04-03 at 06:57 +0800, chen sui wrote:
> >>> Hello SWIFT,
> >>> I found a solution to the problem. It seems to be working for me in this specific situation so far.
> >>> The solution is to comment out lines 242 through 253 in file "cog/modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java" and recompile the JAR file "cog-provider-localscheduler-0.4.jar".
> >>> After this change, PBSExecutor would no longer insert the starred lines and it now works on the cluster I'm using
> >>> Hope this may be useful for people who encounter the same issue.
> >>> Thanks!Tommy Chen
> >>>
> >>> From: baiypwup at hotmail.com
> >>> To: swift-user at ci.uchicago.edu
> >>> Subject: SWIFT Generates "incompatible" PBS scripts
> >>> Date: Thu, 3 Apr 2014 01:28:39 +0800
> >>>
> >>>
> >>>
> >>>
> >>> Hello SWIFT,I have been using SWIFT on two different clusters, it has always been running flawlessly on one of the clusters, but it fails on the second one.
> >>>
> >>> I think I have narrowed down the cause of the problem;
> >>>
> >>> When I have these lines (the 2 lines with stars in the following example) in a generated PBS script,
> >>> [begin example]
> >>> #PBS -S /bin/bash
> >>> #PBS -N B0402-0012360-0
> >>> #PBS -m n
> >>> (omitted)
> >>> export WORKER_LOGGING_LEVEL=NONE   (*)
> >>> #PBS -v WORKER_LOGGING_LEVEL  (*)
> >>> [end example]
> >>>
> >>> The last two lines is causing the PBS to be incompatible with one of the clusters I'm using. When I submit such a script, I get the following error.
> >>>
> >>> [begin Console output]
> >>> [cs900601 at eric2 scripts]$ qsub PBS4684660157119642383.submit
> >>> 506530.eric2
> >>> ** Job deleted, insufficient hours (0 < 96) left in allocation null **
> >>> [end Console output]
> >>>
> >>> Clearly, for some reason the qsub on this cluster has "forgotten" every "#PBS" directive prior to the "export" line.
> >>>
> >>> The reason seems to be
> >>>      1) the cluster I'm using does not support the "-v" switch.
> >>>      2) Does not accept PBS script whose "#PBS" lines have been "interrupted" by an non PBS directive (the "export" above).
> >>>
> >>> That means, if I am able to make Swift remove the 2 lines during script generation the problem would go away.
> >>>
> >>> So my question is, is it possible to do this through Swift configurations?
> >>> I have searched the configuration files but not seem to find a way to do it. The generated script says in the 1st line it's generated by a Java class "by class: class org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor".
> >>>
> >>> Is there a way to check out the source of that "PBSExecutor" class to see what is adding these two lines with asterisks in the generated script and disable it?
> >>>
> >>> Thanks!Tommy Chen
> >>>    		 	   		   		 	   		
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 




From gmgall at lncc.br  Thu Apr  3 19:18:13 2014
From: gmgall at lncc.br (Guilherme Gall)
Date: Thu, 03 Apr 2014 21:18:13 -0300
Subject: [Swift-user] Converting sites configuration
Message-ID: <533DFA45.8030809@lncc.br>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

I'm trying to configure the sites that I intend to use to run jobs
with the new style of configuration that centralizes the sites
configuration in the swift.properties file [1].

The localhost worked fine just setting initialScore, jobManager,
filesystem and workdir directives. But I'm having troubles configuring
Swift to use a local SGE queue.

Is there any documentation available explaining how to convert the
sites.xml style to the new style? I would like to centralize
everything in the swift.properties file.

I know that the following sites.xml works fine:

- -----
<config>
  <pool handle="sunhpc">
    <gridftp url="local://localhost" />
    <!-- <execution provider="sge" url="localhost" /> -->
    <execution provider="coaster" jobmanager="local:sge"
url="localhost" />
    <profile namespace="globus" key="pe">mpi</profile>
    <profile namespace="globus" key="queue">linux.q</profile>
    <profile key="jobThrottle" namespace="karajan">1.27</profile>
    <profile namespace="karajan" key="initialScore">624</profile>
    <filesystem provider="local" url="none" />
    <workdirectory>/prj/cenapadrj/gmgall/swiftwork</workdirectory>
  </pool>
</config>
- -----

But I wasn't able to put this directives in the swift.properties file.

[1]
http://swift-lang.org/guides/trunk/userguide/userguide.html#_grouping_site_properties


Thanks in advance,
- -- 
Guilherme Gall
CSR/LNCC
GPG Public Key ID: A65ED0D5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTPfpFAAoJEG9WBlOmXtDVocwH/1RvBYXve7MqWH4FkzMMQn3r
29/kysBeTHSFOQRv1FvFeiNQE1N6Mq04eGboTwvdnljWU73b0wz/Nnsh/bSuuYyP
gGYpfyF9/kBkX6/tezjZqHV8M/3ltyrS2pI3HROjTgfCWqV0LHOVo2c2Qyw3pDM0
apIFJu72cBS8VLcRudOCLn/EaTj/Yrj625S2tmwbt6hbIhelTfI/HmhxYVfYVYK9
tPstRROfGqxBlFO2Qldakrh8yl33WgeR/4/1WDtmztrEfyxYHn/lRVzJ6lP7LThA
TM8vYtFObGNmfJKmp7BvuHa55sVMA1JR0zvaacBNcgGmI0J9+lFxNuKWq1g66Tc=
=Am59
-----END PGP SIGNATURE-----


From davidkelly at uchicago.edu  Fri Apr  4 00:18:15 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Fri, 4 Apr 2014 00:18:15 -0500
Subject: [Swift-user] Converting sites configuration
In-Reply-To: <533DFA45.8030809@lncc.br>
References: <533DFA45.8030809@lncc.br>
Message-ID: <CA+_+Ey8YM1-RBqSi6Uzmke0gMMBzJqvAze2yJq2EwTZ38O2zYw@mail.gmail.com>

Hi Guilherme,

There was one setting (pe) that was undocumented and not yet in the new
config. I've just added this and updated the user guide. I've attached a
swift.properties file that I believe should work for you with the latest
0.95 and trunk. Please let me know if you have any issues.

I will add something to the user guide soon that makes it easier to
translate settings directly from sites.xml to the new swift.properties
format.

Thanks,
David


On Thu, Apr 3, 2014 at 7:18 PM, Guilherme Gall <gmgall at lncc.br> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi everyone,
>
> I'm trying to configure the sites that I intend to use to run jobs
> with the new style of configuration that centralizes the sites
> configuration in the swift.properties file [1].
>
> The localhost worked fine just setting initialScore, jobManager,
> filesystem and workdir directives. But I'm having troubles configuring
> Swift to use a local SGE queue.
>
> Is there any documentation available explaining how to convert the
> sites.xml style to the new style? I would like to centralize
> everything in the swift.properties file.
>
> I know that the following sites.xml works fine:
>
> - -----
> <config>
>   <pool handle="sunhpc">
>     <gridftp url="local://localhost" />
>     <!-- <execution provider="sge" url="localhost" /> -->
>     <execution provider="coaster" jobmanager="local:sge"
> url="localhost" />
>     <profile namespace="globus" key="pe">mpi</profile>
>     <profile namespace="globus" key="queue">linux.q</profile>
>     <profile key="jobThrottle" namespace="karajan">1.27</profile>
>     <profile namespace="karajan" key="initialScore">624</profile>
>     <filesystem provider="local" url="none" />
>     <workdirectory>/prj/cenapadrj/gmgall/swiftwork</workdirectory>
>   </pool>
> </config>
> - -----
>
> But I wasn't able to put this directives in the swift.properties file.
>
> [1]
>
> http://swift-lang.org/guides/trunk/userguide/userguide.html#_grouping_site_properties
>
>
> Thanks in advance,
> - --
> Guilherme Gall
> CSR/LNCC
> GPG Public Key ID: A65ED0D5
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTPfpFAAoJEG9WBlOmXtDVocwH/1RvBYXve7MqWH4FkzMMQn3r
> 29/kysBeTHSFOQRv1FvFeiNQE1N6Mq04eGboTwvdnljWU73b0wz/Nnsh/bSuuYyP
> gGYpfyF9/kBkX6/tezjZqHV8M/3ltyrS2pI3HROjTgfCWqV0LHOVo2c2Qyw3pDM0
> apIFJu72cBS8VLcRudOCLn/EaTj/Yrj625S2tmwbt6hbIhelTfI/HmhxYVfYVYK9
> tPstRROfGqxBlFO2Qldakrh8yl33WgeR/4/1WDtmztrEfyxYHn/lRVzJ6lP7LThA
> TM8vYtFObGNmfJKmp7BvuHa55sVMA1JR0zvaacBNcgGmI0J9+lFxNuKWq1g66Tc=
> =Am59
> -----END PGP SIGNATURE-----
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140404/d07c503f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: swift.properties
Type: application/octet-stream
Size: 160 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140404/d07c503f/attachment.obj>

From gmgall at lncc.br  Thu Apr 10 14:08:17 2014
From: gmgall at lncc.br (Guilherme Gall)
Date: Thu, 10 Apr 2014 16:08:17 -0300
Subject: [Swift-user] Converting sites configuration
In-Reply-To: <CA+_+Ey8YM1-RBqSi6Uzmke0gMMBzJqvAze2yJq2EwTZ38O2zYw@mail.gmail.com>
References: <533DFA45.8030809@lncc.br>
	<CA+_+Ey8YM1-RBqSi6Uzmke0gMMBzJqvAze2yJq2EwTZ38O2zYw@mail.gmail.com>
Message-ID: <5346EC21.9040209@lncc.br>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Em 04-04-2014 02:18, David Kelly escreveu:
> Hi Guilherme,
> 
> There was one setting (pe) that was undocumented and not yet in the
> new config. I've just added this and updated the user guide. I've
> attached a swift.properties file that I believe should work for you
> with the latest 0.95 and trunk. Please let me know if you have any
> issues.
> 
> I will add something to the user guide soon that makes it easier
> to translate settings directly from sites.xml to the new
> swift.properties format.
> 
> Thanks, David
> 
> 

Hi David,

The swift.properties that you sent me seems to work.

But now I'm having another problem that wasn't happening before.
Follows the error message that I receive sometimes:

- ----------
RunID: run011
Progress: Qui, 10 abr 2014 14:58:18-0300
Progress: Qui, 10 abr 2014 14:58:19-0300  Stage in:1
Progress: Qui, 10 abr 2014 14:58:20-0300  Active:1

Execution failed:
Exception in generate_requests:
    Arguments: [-o, teste.txt, --min_occurrences, 10, --map_list,
workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio12.asc,workshop/Brasil_ASC/bio2.asc,
- -m, workshop/Brasil_ASC/bio9.asc, --output_format_file,
workshop/Brasil_ASC/bio10.asc, --output_map_list,
workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio3.asc,
- --output_mask_file, workshop/Brasil_ASC/bio8.asc]
    Host: sunhpc
    Directory:
workflow-openmodeller-run011/jobs/b/generate_requests-b6u6c1pl
        exception @ swift-int.k, line: 520
Caused by: Failed to link input file workshop/Brasil_ASC/alt.asc
        throw @ swift-int.k, line: 68

        k:assign @ swift.k, line: 194
Caused by: Exception in generate_requests:
    Arguments: [-o, teste.txt, --min_occurrences, 10, --map_list,
workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio12.asc,workshop/Brasil_ASC/bio2.asc,
- -m, workshop/Brasil_ASC/bio9.asc, --output_format_file,
workshop/Brasil_ASC/bio10.asc, --output_map_list,
workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio3.asc,
- --output_mask_file, workshop/Brasil_ASC/bio8.asc]
    Host: sunhpc
    Directory:
workflow-openmodeller-run011/jobs/b/generate_requests-b6u6c1pl
        exception @ swift-int.k, line: 520
Caused by: Failed to link input file workshop/Brasil_ASC/alt.asc
        throw @ swift-int.k, line: 68

- ----------

This is happening in 2 different environments with Swift installed.
Both with recent code from trunk.  One of them is using revision 7758
and another 7759 (both from April 2014).

There is a third machine using Swift revision 7575 that works
perfectly. That was the environment that I used to develop my Swift
script[1]. It always worked fine there.

Is there the possibility of a recent change has generated a bug?

Best regasrds,

[1]
https://github.com/sibbr/sdm-workflows/blob/master/workflow-openmodeller.swift
- -- 
Guilherme Gall
CSR/LNCC
GPG Public Key ID: A65ED0D5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTRuwhAAoJEG9WBlOmXtDV81EH/2r730TDR4bzc/s+F/jHUme5
TR8Muw4bQcg/pY1gh7/yBkIXFjt6g9rc0nWEyYgWCK+dYImig1BfVqTQCu6u5xoZ
pse4hRCC+Tsg+isJGfz5gKst5MQjXDjmE8Cy4Aw57qmlO/fPDGxXEo5ALkvJI9tZ
NIGFygvp8vPsZ9BnfEY+qeoSMlFW+EGWQpfIVMKEmnZ1rKBQKQ64pf5ATXa6v9aL
+koBJkeC5H2M0uZ9OEgfbLLJQvNiJ7fEQvlVQ7WdwNNEUWp0qLZdpA/aZqr8Lo8x
EiLW/FFc3ISOuQGSsKWyM9EPOYgBXT9HDZLx19GTtckyW53VKQCVrnOf822Q86k=
=iUZP
-----END PGP SIGNATURE-----


From davidkelly at uchicago.edu  Mon Apr 14 13:13:39 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Mon, 14 Apr 2014 13:13:39 -0500
Subject: [Swift-user] Converting sites configuration
In-Reply-To: <5346EC21.9040209@lncc.br>
References: <533DFA45.8030809@lncc.br>
	<CA+_+Ey8YM1-RBqSi6Uzmke0gMMBzJqvAze2yJq2EwTZ38O2zYw@mail.gmail.com>
	<5346EC21.9040209@lncc.br>
Message-ID: <CA+_+Ey_j02CCqqT2ZtZeuKBtK3jap8Cr+gKY181gSGMvAkxySQ@mail.gmail.com>

Hi Guilherme,

I'm not immediately sure it's failing to link inputs. Could you please send
a log file from when this happens so I can take a closer look?

Thanks,
David



On Thu, Apr 10, 2014 at 2:08 PM, Guilherme Gall <gmgall at lncc.br> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Em 04-04-2014 02:18, David Kelly escreveu:
> > Hi Guilherme,
> >
> > There was one setting (pe) that was undocumented and not yet in the
> > new config. I've just added this and updated the user guide. I've
> > attached a swift.properties file that I believe should work for you
> > with the latest 0.95 and trunk. Please let me know if you have any
> > issues.
> >
> > I will add something to the user guide soon that makes it easier
> > to translate settings directly from sites.xml to the new
> > swift.properties format.
> >
> > Thanks, David
> >
> >
>
> Hi David,
>
> The swift.properties that you sent me seems to work.
>
> But now I'm having another problem that wasn't happening before.
> Follows the error message that I receive sometimes:
>
> - ----------
> RunID: run011
> Progress: Qui, 10 abr 2014 14:58:18-0300
> Progress: Qui, 10 abr 2014 14:58:19-0300  Stage in:1
> Progress: Qui, 10 abr 2014 14:58:20-0300  Active:1
>
> Execution failed:
> Exception in generate_requests:
>     Arguments: [-o, teste.txt, --min_occurrences, 10, --map_list,
>
> workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio12.asc,workshop/Brasil_ASC/bio2.asc,
> - -m, workshop/Brasil_ASC/bio9.asc, --output_format_file,
> workshop/Brasil_ASC/bio10.asc, --output_map_list,
> workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio3.asc,
> - --output_mask_file, workshop/Brasil_ASC/bio8.asc]
>     Host: sunhpc
>     Directory:
> workflow-openmodeller-run011/jobs/b/generate_requests-b6u6c1pl
>         exception @ swift-int.k, line: 520
> Caused by: Failed to link input file workshop/Brasil_ASC/alt.asc
>         throw @ swift-int.k, line: 68
>
>         k:assign @ swift.k, line: 194
> Caused by: Exception in generate_requests:
>     Arguments: [-o, teste.txt, --min_occurrences, 10, --map_list,
>
> workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio12.asc,workshop/Brasil_ASC/bio2.asc,
> - -m, workshop/Brasil_ASC/bio9.asc, --output_format_file,
> workshop/Brasil_ASC/bio10.asc, --output_map_list,
> workshop/Brasil_ASC/alt.asc,workshop/Brasil_ASC/bio3.asc,
> - --output_mask_file, workshop/Brasil_ASC/bio8.asc]
>     Host: sunhpc
>     Directory:
> workflow-openmodeller-run011/jobs/b/generate_requests-b6u6c1pl
>         exception @ swift-int.k, line: 520
> Caused by: Failed to link input file workshop/Brasil_ASC/alt.asc
>         throw @ swift-int.k, line: 68
>
> - ----------
>
> This is happening in 2 different environments with Swift installed.
> Both with recent code from trunk.  One of them is using revision 7758
> and another 7759 (both from April 2014).
>
> There is a third machine using Swift revision 7575 that works
> perfectly. That was the environment that I used to develop my Swift
> script[1]. It always worked fine there.
>
> Is there the possibility of a recent change has generated a bug?
>
> Best regasrds,
>
> [1]
>
> https://github.com/sibbr/sdm-workflows/blob/master/workflow-openmodeller.swift
> - --
> Guilherme Gall
> CSR/LNCC
> GPG Public Key ID: A65ED0D5
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTRuwhAAoJEG9WBlOmXtDV81EH/2r730TDR4bzc/s+F/jHUme5
> TR8Muw4bQcg/pY1gh7/yBkIXFjt6g9rc0nWEyYgWCK+dYImig1BfVqTQCu6u5xoZ
> pse4hRCC+Tsg+isJGfz5gKst5MQjXDjmE8Cy4Aw57qmlO/fPDGxXEo5ALkvJI9tZ
> NIGFygvp8vPsZ9BnfEY+qeoSMlFW+EGWQpfIVMKEmnZ1rKBQKQ64pf5ATXa6v9aL
> +koBJkeC5H2M0uZ9OEgfbLLJQvNiJ7fEQvlVQ7WdwNNEUWp0qLZdpA/aZqr8Lo8x
> EiLW/FFc3ISOuQGSsKWyM9EPOYgBXT9HDZLx19GTtckyW53VKQCVrnOf822Q86k=
> =iUZP
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140414/b9374620/attachment.html>

From gmgall at lncc.br  Thu Apr 17 09:28:42 2014
From: gmgall at lncc.br (Guilherme Gall)
Date: Thu, 17 Apr 2014 11:28:42 -0300
Subject: [Swift-user] Converting sites configuration
In-Reply-To: <CA+_+Ey_j02CCqqT2ZtZeuKBtK3jap8Cr+gKY181gSGMvAkxySQ@mail.gmail.com>
References: <533DFA45.8030809@lncc.br>	<CA+_+Ey8YM1-RBqSi6Uzmke0gMMBzJqvAze2yJq2EwTZ38O2zYw@mail.gmail.com>	<5346EC21.9040209@lncc.br>
	<CA+_+Ey_j02CCqqT2ZtZeuKBtK3jap8Cr+gKY181gSGMvAkxySQ@mail.gmail.com>
Message-ID: <534FE51A.9070704@lncc.br>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Em 14-04-2014 15:13, David Kelly escreveu:
> Hi Guilherme,
> 
> I'm not immediately sure it's failing to link inputs. Could you
> please send a log file from when this happens so I can take a
> closer look?
> 
> Thanks, David
> 
> 

Hello David,

I'm sending the logs attached. The same Swift script with the same
input files works fine in another machine with revision 7575.

Best regards,
- -- 
Guilherme Gall
CSR/LNCC
GPG Public Key ID: A65ED0D5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTT+UXAAoJEG9WBlOmXtDVAX8H/0pBTd7D/JI5MYabxzvZB7ta
943lVXxo0qpFftF4Fq/aVfLY2QedA5ZQl0o64trI+4D9tgdd3nIm+XBmr6rk6nuZ
/k0psCWyf5zYmv52VCp2AUYti/H5435GzMQTf+4sTRoO+/q2SvQ3fMEttSzyUNop
OqBEaMRDCAJku/o1CzocgoU3Yys691AHu1uyd6a4duAZAYVK3y2Oqp0zlFh3x32l
Gx1+8W2DSlESzCvMQN8TZBK0S//hb8IwyEX4NEe2u7aR55PYFafRcorRsJqXwIkP
sNppbTxBYSwYoIZzkvHV3m8ZJ44F7bJbQzvHF7pOuNAhuozBPtL7NlpMlk2v8qQ=
=luzb
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run001.tar.gz
Type: application/gzip
Size: 10678 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20140417/b41b2df7/attachment.bin>