From hategan at mcs.anl.gov  Fri Jul  1 13:28:13 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 01 Jul 2011 13:28:13 -0500
Subject: [Swift-devel] plans
Message-ID: <1309544893.15627.8.camel@blabla>

In a meeting yesterday we discussed some of the swift plans for the rest
of the summer.

The basic ideas were:
- branch the current trunk out to 0.93
  * we should do this in the next couple of weeks. There are a few
deadlocks that I want to fix in trunk before we do that.
  * 0.93 would be the first release to include the stuff in the fast
branch
- the main goals for 0.94 would be:
  * Yadu's array key typing
  * garbage collection (both memory and data mapped with the concurrent
mapper)
  * throttling based on the exact number of workers available when using
passive coaster workers
  * possibly a set of plotting tools based on the infrastructure behind
the TUI.
  * these would initially go into trunk after the 0.93 branching.

Mike, Justin, please fill in what I missed.

Mihael




From ketancmaheshwari at gmail.com  Fri Jul  1 13:37:16 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Fri, 1 Jul 2011 13:37:16 -0500
Subject: [Swift-devel] plans
In-Reply-To: <1309544893.15627.8.camel@blabla>
References: <1309544893.15627.8.camel@blabla>
Message-ID: <BANLkTik93kgkKsxMjf_23EhsrrpZumLj3w@mail.gmail.com>

We may also want to fix the suspected hang-checker-behavior possibly causing
stalls for pbs provider on Beagle. I am planning to do some tests and see if
indeed the cause is what I am suspecting.

On Fri, Jul 1, 2011 at 1:28 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> In a meeting yesterday we discussed some of the swift plans for the rest
> of the summer.
>
> The basic ideas were:
> - branch the current trunk out to 0.93
>  * we should do this in the next couple of weeks. There are a few
> deadlocks that I want to fix in trunk before we do that.
>  * 0.93 would be the first release to include the stuff in the fast
> branch
> - the main goals for 0.94 would be:
>  * Yadu's array key typing
>  * garbage collection (both memory and data mapped with the concurrent
> mapper)
>  * throttling based on the exact number of workers available when using
> passive coaster workers
>  * possibly a set of plotting tools based on the infrastructure behind
> the TUI.
>  * these would initially go into trunk after the 0.93 branching.
>
> Mike, Justin, please fill in what I missed.
>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110701/42bb306c/attachment.html>

From hategan at mcs.anl.gov  Fri Jul  1 13:54:14 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 01 Jul 2011 13:54:14 -0500
Subject: [Swift-devel] plans
In-Reply-To: <BANLkTik93kgkKsxMjf_23EhsrrpZumLj3w@mail.gmail.com>
References: <1309544893.15627.8.camel@blabla>
	<BANLkTik93kgkKsxMjf_23EhsrrpZumLj3w@mail.gmail.com>
Message-ID: <1309546454.16953.2.camel@blabla>

On Fri, 2011-07-01 at 13:37 -0500, Ketan Maheshwari wrote:
> We may also want to fix the suspected hang-checker-behavior possibly
> causing stalls for pbs provider on Beagle.

Of course. The purpose of the release branch is to isolate release code
from potentially buggy code (trunk). It would then be tested and fixed
as needed.




From hategan at mcs.anl.gov  Fri Jul  1 18:12:31 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 01 Jul 2011 18:12:31 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
Message-ID: <1309561951.30533.1.camel@blabla>

On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
> Hello,
>     I am going to rename readData2 to something more meaningful.  I am
> thinking maybe readStructured?  I will keep readData2 as an alias to
> the renamed function as too not break anyones current code but the
> userguide will reflect the renamed function.  Any suggestions on the
> name or is readStructured good?

I think we should drop readData and slowly replace it with readData2.
readStructured doesn't seem to be more illuminating to me.




From jonmon at utexas.edu  Fri Jul  1 18:22:23 2011
From: jonmon at utexas.edu (Jonathan S Monette)
Date: Fri, 1 Jul 2011 18:22:23 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <1309561951.30533.1.camel@blabla>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
Message-ID: <BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>

I am all for dropping readData. writeData would have to be modified to
output data for "readData2".

Do you have a better name? I thought maybe readMapping or
readExplicitMapping but Justin said the mapping term maybe should not be
used in this case.
On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
>> Hello,
>> I am going to rename readData2 to something more meaningful. I am
>> thinking maybe readStructured? I will keep readData2 as an alias to
>> the renamed function as too not break anyones current code but the
>> userguide will reflect the renamed function. Any suggestions on the
>> name or is readStructured good?
>
> I think we should drop readData and slowly replace it with readData2.
> readStructured doesn't seem to be more illuminating to me.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110701/34386597/attachment.html>

From skenny at uchicago.edu  Fri Jul  1 20:04:08 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Fri, 1 Jul 2011 18:04:08 -0700
Subject: [Swift-devel] readData2
In-Reply-To: <BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
Message-ID: <BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>

my 2 cents...readStruct...slightly more succinct :)

On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette <jonmon at utexas.edu>wrote:

> I am all for dropping readData. writeData would have to be modified to
> output data for "readData2".
>
> Do you have a better name? I thought maybe readMapping or
> readExplicitMapping but Justin said the mapping term maybe should not be
> used in this case.
> On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov> wrote:
> > On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
> >> Hello,
> >> I am going to rename readData2 to something more meaningful. I am
> >> thinking maybe readStructured? I will keep readData2 as an alias to
> >> the renamed function as too not break anyones current code but the
> >> userguide will reflect the renamed function. Any suggestions on the
> >> name or is readStructured good?
> >
> > I think we should drop readData and slowly replace it with readData2.
> > readStructured doesn't seem to be more illuminating to me.
> >
> >
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110701/7dc972a4/attachment.html>

From hategan at mcs.anl.gov  Fri Jul  1 20:09:44 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 01 Jul 2011 20:09:44 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
Message-ID: <1309568984.3651.0.camel@blabla>

On Fri, 2011-07-01 at 18:04 -0700, Sarah Kenny wrote:
> my 2 cents...readStruct...slightly more succinct :)

2 more cents: read. Can't get much better than that.
> 
> On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette <jonmon at utexas.edu>
> wrote:
>         I am all for dropping readData. writeData would have to be
>         modified to output data for "readData2". 
>         
>         Do you have a better name? I thought maybe readMapping or
>         readExplicitMapping but Justin said the mapping term maybe
>         should not be used in this case. 
>         
>         
>         On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov>
>         wrote:
>         > On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
>         >> Hello,
>         >> I am going to rename readData2 to something more
>         meaningful. I am
>         >> thinking maybe readStructured? I will keep readData2 as an
>         alias to
>         >> the renamed function as too not break anyones current code
>         but the
>         >> userguide will reflect the renamed function. Any
>         suggestions on the
>         >> name or is readStructured good?
>         > 
>         > I think we should drop readData and slowly replace it with
>         readData2.
>         > readStructured doesn't seem to be more illuminating to me.
>         > 
>         > 
>         
>         
>         _______________________________________________
>         Swift-devel mailing list
>         Swift-devel at ci.uchicago.edu
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
> 
> 
> 
> -- 
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
> 




From tim.g.armstrong at gmail.com  Fri Jul  1 20:12:55 2011
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Fri, 1 Jul 2011 18:12:55 -0700
Subject: [Swift-devel] readData2
In-Reply-To: <1309568984.3651.0.camel@blabla>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
Message-ID: <BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>

You could go full unix shell style and call it "rd"

On Fri, Jul 1, 2011 at 6:09 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Fri, 2011-07-01 at 18:04 -0700, Sarah Kenny wrote:
> > my 2 cents...readStruct...slightly more succinct :)
>
> 2 more cents: read. Can't get much better than that.
> >
> > On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette <jonmon at utexas.edu>
> > wrote:
> >         I am all for dropping readData. writeData would have to be
> >         modified to output data for "readData2".
> >
> >         Do you have a better name? I thought maybe readMapping or
> >         readExplicitMapping but Justin said the mapping term maybe
> >         should not be used in this case.
> >
> >
> >         On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov>
> >         wrote:
> >         > On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
> >         >> Hello,
> >         >> I am going to rename readData2 to something more
> >         meaningful. I am
> >         >> thinking maybe readStructured? I will keep readData2 as an
> >         alias to
> >         >> the renamed function as too not break anyones current code
> >         but the
> >         >> userguide will reflect the renamed function. Any
> >         suggestions on the
> >         >> name or is readStructured good?
> >         >
> >         > I think we should drop readData and slowly replace it with
> >         readData2.
> >         > readStructured doesn't seem to be more illuminating to me.
> >         >
> >         >
> >
> >
> >         _______________________________________________
> >         Swift-devel mailing list
> >         Swift-devel at ci.uchicago.edu
> >
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> >
> >
> > --
> > Sarah Kenny
> > Programmer
> > University of Chicago, Computation Institute
> > University of California Irvine, Dept. of Neurology
> > 773-818-8300
> >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110701/0eb1da74/attachment.html>

From wozniak at mcs.anl.gov  Fri Jul  1 21:04:03 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Fri, 1 Jul 2011 21:04:03 -0500 (Central Daylight Time)
Subject: [Swift-devel] readData2
In-Reply-To: <BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
Message-ID: <alpine.WNT.2.00.1107012055280.912@justinwozniak>


fopen(3) is smart enough to know that "r" means read.  Certainly Swift is 
smarter than fopen.

However, I think there is a lot of value in allowing users to read 
unformatted, "one value per line" data and I don't think we should force 
users to convert possibly large text files into Swift format.  So, I think 
there is value in keeping the two functionalities separate.

If we're going to mandate a file format, we should try to be compatible 
with JSON or something.  But I don't think we should mandate a file 
format.

 	Justin

On Fri, 1 Jul 2011, Tim Armstrong wrote:

> You could go full unix shell style and call it "rd"
>
> On Fri, Jul 1, 2011 at 6:09 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>
>> On Fri, 2011-07-01 at 18:04 -0700, Sarah Kenny wrote:
>>> my 2 cents...readStruct...slightly more succinct :)
>>
>> 2 more cents: read. Can't get much better than that.
>>>
>>> On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette <jonmon at utexas.edu>
>>> wrote:
>>>         I am all for dropping readData. writeData would have to be
>>>         modified to output data for "readData2".
>>>
>>>         Do you have a better name? I thought maybe readMapping or
>>>         readExplicitMapping but Justin said the mapping term maybe
>>>         should not be used in this case.
>>>
>>>
>>>         On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov>
>>>         wrote:
>>>        > On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
>>>        >> Hello,
>>>        >> I am going to rename readData2 to something more
>>>         meaningful. I am
>>>        >> thinking maybe readStructured? I will keep readData2 as an
>>>         alias to
>>>        >> the renamed function as too not break anyones current code
>>>         but the
>>>        >> userguide will reflect the renamed function. Any
>>>         suggestions on the
>>>        >> name or is readStructured good?
>>>        >
>>>        > I think we should drop readData and slowly replace it with
>>>         readData2.
>>>        > readStructured doesn't seem to be more illuminating to me.
>>>        >
>>>        >
>>>
>>>
>>>         _______________________________________________
>>>         Swift-devel mailing list
>>>         Swift-devel at ci.uchicago.edu
>>>
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>>>
>>>
>>> --
>>> Sarah Kenny
>>> Programmer
>>> University of Chicago, Computation Institute
>>> University of California Irvine, Dept. of Neurology
>>> 773-818-8300
>>>
>>
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Sat Jul  2 13:09:18 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 02 Jul 2011 13:09:18 -0500
Subject: [Swift-devel] test suite
Message-ID: <1309630158.28709.7.camel@blabla>

I've been playing with the test suite a bit, and I have a few comments:

- Ben's ./run scripts had this nice feature which allowed you to resume
a test run. This was great if you were testing some changes and wanted
to go through all tests and fix failing ones, and continue without
having to repeatedly re-run the previous tests. Ben's scripts don't work
any more, and the new test suite doesn't do that. It would be nice if
either was to work.

- it would be nice to have a way where you just run all the local tests
straight from the checkout with minimal intervention (i.e. run whatever
swift is in my path). Out of the box, ./suite.sh
-t /groups/group-all-local.sh doesn't work because it cannot find my
swift directory. I've changed the following to make that work:
TOPDIR=`readlink -f $PWD/../../../..`

Mihael



From jonmon at utexas.edu  Sat Jul  2 14:51:47 2011
From: jonmon at utexas.edu (Jonathan S Monette)
Date: Sat, 2 Jul 2011 14:51:47 -0500
Subject: [Swift-devel] readData2
Message-ID: <BANLkTim8_HYv8gb+ZO3tAWJ3325H9fjnNw@mail.gmail.com>

I think having Swift follow a more "universal"(I don't know how universal
JSON is but I figure it is used a lot) format like JSON has its merits and
maybe Swift should be able to read this format.

I also believe the the readData format should be kept around.
 On Jul 1, 2011 9:04 PM, "Justin M Wozniak" <wozniak at mcs.anl.gov> wrote:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110702/72268026/attachment.html>

From hategan at mcs.anl.gov  Sat Jul  2 21:14:15 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 02 Jul 2011 21:14:15 -0500
Subject: [Swift-devel] the wrapper map
Message-ID: <1309659255.8876.4.camel@blabla>

So you are going to hate me, but I got rid of the wrapper map which was
a way of wrapping swift data in karajan futures. The sad part about it
is that it required a global lock to be held for each access to some
future, which meant both slowdowns and the potential for deadlocks that
were getting annoying to fix.

It's now gone and each piece of swift data has its own future if it's
needed. That bad part of that is the large number of source files that
needed to be changed. The good part about it is that some of them became
simpler.

So if you're working on trunk code, I'd suggest doing an update sooner
rather than later.

Mihael



From wozniak at mcs.anl.gov  Sun Jul  3 17:05:56 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Sun, 3 Jul 2011 17:05:56 -0500 (Central Daylight Time)
Subject: [Swift-devel] test suite
In-Reply-To: <1309630158.28709.7.camel@blabla>
References: <1309630158.28709.7.camel@blabla>
Message-ID: <alpine.WNT.2.00.1107031703580.4672@JWOZNIAK-DESK>


The closest thing we have to test suite restart is the manual use of -k/-n 
to skip ahead and exit early.

The existing source tree can be specified with the -o option.

 	Justin

On Sat, 2 Jul 2011, Mihael Hategan wrote:

> I've been playing with the test suite a bit, and I have a few comments:
>
> - Ben's ./run scripts had this nice feature which allowed you to resume
> a test run. This was great if you were testing some changes and wanted
> to go through all tests and fix failing ones, and continue without
> having to repeatedly re-run the previous tests. Ben's scripts don't work
> any more, and the new test suite doesn't do that. It would be nice if
> either was to work.
>
> - it would be nice to have a way where you just run all the local tests
> straight from the checkout with minimal intervention (i.e. run whatever
> swift is in my path). Out of the box, ./suite.sh
> -t /groups/group-all-local.sh doesn't work because it cannot find my
> swift directory. I've changed the following to make that work:
> TOPDIR=`readlink -f $PWD/../../../..`
>
> Mihael
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>

-- 
Justin M Wozniak


From davidkelly999 at gmail.com  Sun Jul  3 20:40:47 2011
From: davidkelly999 at gmail.com (David Kelly)
Date: Sun, 3 Jul 2011 20:40:47 -0500
Subject: [Swift-devel] Swift unresponsive while using local provider.
In-Reply-To: <1308853713.17296.0.camel@blabla>
References: <BAY156-w505A430E3126E1F6F806199C6D0@phx.gbl>
	<1057363198.21380.1308339533392.JavaMail.root@zimbra.anl.gov>
	<BANLkTikEaYj8gJjz_p0OJQCNVpevGMTV5Q@mail.gmail.com>
	<1308340585.14231.0.camel@blabla>
	<BAY156-w65F158DA12C1A99E42124D9C6D0@phx.gbl>
	<1308355792.24489.2.camel@blabla>
	<BAY156-w38F63744968EC2E4ED5ED59C6C0@phx.gbl>
	<1308356386.24582.2.camel@blabla>
	<BAY156-w64DC1F7C60988A1D590FB9C6C0@phx.gbl>
	<1308358465.24760.10.camel@blabla>
	<BAY156-w13B07C2D9777E611BF7BED9C6C0@phx.gbl>
	<1308367911.25750.0.camel@blabla>
	<BAY156-w16A4A1FB055839CE62C7F09C6C0@phx.gbl>
	<1308368849.25915.3.camel@blabla>
	<BAY156-w49B30735AD916251ADB22F9C6C0@phx.gbl>
	<BANLkTim0vb47kjfqwfh65HY1D8ELhi9Hdw@mail.gmail.com>
	<BAY156-w442D26F6D4F07074EB4B649C6C0@phx.gbl>
	<1308370144.26103.1.camel@blabla>
	<BAY156-w8529C910F18014CF13E179C6C0@phx.gbl>
	<1308426730.30334.0.camel@blabla>
	<BANLkTikQ4KaAWhc805=MPKHXhbA_tDPVvQ@mail.gmail.com>
	<1308853713.17296.0.camel@blabla>
Message-ID: <CALwLSKoLqnWo5a+nqy-gevFC1F2V7usQVxUZOwNrpMVg_JSQzA@mail.gmail.com>

This seems to be fixed now with the latest version. The test script ran
successfully around 2000 times before I killed it.

David

On Thu, Jun 23, 2011 at 1:28 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> I committed a tentative fix to svn. swift trunk r4666.
>
> On Sat, 2011-06-18 at 21:21 -0500, David Kelly wrote:
> > Here's one I got with the latest version tonight:
> >
> > 2011-06-18 21:01:34
> > Full thread dump Java HotSpot(TM) Server VM (19.1-b02 mixed mode):
> >
> > "Attach Listener" daemon prio=10 tid=0x087d6c00 nid=0x882 runnable
> > [0x00000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Progress ticker" daemon prio=10 tid=0x9e854400 nid=0x85e waiting on
> > condition [0x9dfad000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> >     at java.lang.Thread.sleep(Native Method)
> >     at org.griphyn.vdl.karajan.lib.RuntimeStats
> > $ProgressTicker.run(RuntimeStats.java:141)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Restart Log Sync" daemon prio=10 tid=0x9e82bc00 nid=0x85d in
> > Object.wait() [0x9dffe000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0xaedb4778> (a
> > org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread)
> >     at java.lang.Object.wait(Object.java:485)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.run(SyncThread.java:47)
> >     - locked <0xaedb4778> (a
> > org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Overloaded Host Monitor" daemon prio=10 tid=0x08b19400 nid=0x85c
> > waiting on condition [0x9e15c000]
> >    java.lang.Thread.State: TIMED_WAITING (sleeping)
> >     at java.lang.Thread.sleep(Native Method)
> >     at
> >
> org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run(OverloadedHostMonitor.java:47)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Timer-0" daemon prio=10 tid=0x08354400 nid=0x85b in Object.wait()
> > [0x9e1ad000]
> >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0xaf5e2c38> (a java.util.TaskQueue)
> >     at java.util.TimerThread.mainLoop(Timer.java:509)
> >     - locked <0xaf5e2c38> (a java.util.TaskQueue)
> >     at java.util.TimerThread.run(Timer.java:462)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "NBS0" daemon prio=10 tid=0x087de000 nid=0x85a waiting on condition
> > [0x9e1fe000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5e3628> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-8" prio=10 tid=0x08445c00 nid=0x859 waiting for monitor
> > entry [0x9e369000]
> >    java.lang.Thread.State: BLOCKED (on object monitor)
> >     at org.griphyn.vdl.karajan.WrapperMap.close(WrapperMap.java:25)
> >     - waiting to lock <0xaf5ed6c8> (a
> > org.griphyn.vdl.karajan.WrapperMap)
> >     at
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.closeShallow(VDLFunction.java:516)
> >     - locked <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> >
> org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:121)
> >     at
> > org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:49)
> >     - locked <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:71)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
> >     at
> >
> org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227)
> >     at
> > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104)
> >     at
> >
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
> >     at java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:441)
> >     at java.util.concurrent.FutureTask
> > $Sync.innerRun(FutureTask.java:303)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - <0xaf5e3b10> (a java.util.concurrent.locks.ReentrantLock
> > $NonfairSync)
> >
> > "pool-1-thread-7" prio=10 tid=0x08443000 nid=0x858 waiting on
> > condition [0x9e3ba000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-6" prio=10 tid=0x08441800 nid=0x857 waiting for monitor
> > entry [0x9e40b000]
> >    java.lang.Thread.State: BLOCKED (on object monitor)
> >     at
> >
> org.griphyn.vdl.mapping.AbstractDataNode.addListener(AbstractDataNode.java:583)
> >     - waiting to lock <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> >
> org.griphyn.vdl.karajan.DSHandleFutureWrapper.<init>(DSHandleFutureWrapper.java:24)
> >     at
> > org.griphyn.vdl.karajan.WrapperMap.addNodeListener(WrapperMap.java:61)
> >     - locked <0xaf5ed6c8> (a org.griphyn.vdl.karajan.WrapperMap)
> >     at
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.addFutureListener(VDLFunction.java:523)
> >     at org.griphyn.vdl.karajan.lib.Stagein.function(Stagein.java:88)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
> >     at
> >
> org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227)
> >     at
> > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104)
> >     at
> >
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
> >     at java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:441)
> >     at java.util.concurrent.FutureTask
> > $Sync.innerRun(FutureTask.java:303)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - <0xaf5ec3a0> (a java.util.concurrent.locks.ReentrantLock
> > $NonfairSync)
> >
> > "pool-1-thread-5" prio=10 tid=0x085d2000 nid=0x856 waiting on
> > condition [0x9e45c000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-4" prio=10 tid=0x085d2800 nid=0x855 waiting on
> > condition [0x9e4ad000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-3" prio=10 tid=0x08839400 nid=0x854 waiting on
> > condition [0x9e65c000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-2" prio=10 tid=0x08837800 nid=0x853 waiting on
> > condition [0x9e6ad000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "pool-1-thread-1" prio=10 tid=0x9e826c00 nid=0x852 waiting on
> > condition [0x9e6fe000]
> >    java.lang.Thread.State: WAITING (parking)
> >     at sun.misc.Unsafe.park(Native Method)
> >     - parking to wait for  <0xaf5ec2b0> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >     at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >     at java.util.concurrent.locks.AbstractQueuedSynchronizer
> > $ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >     at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:907)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Hang checker" prio=10 tid=0x9e81bc00 nid=0x851 waiting for monitor
> > entry [0x9e4fe000]
> >    java.lang.Thread.State: BLOCKED (on object monitor)
> >     at org.griphyn.vdl.karajan.Monitor.dumpVariables(Monitor.java:220)
> >     - waiting to lock <0xaf5ed6c8> (a
> > org.griphyn.vdl.karajan.WrapperMap)
> >     at org.griphyn.vdl.karajan.HangChecker.run(HangChecker.java:54)
> >     at java.util.TimerThread.mainLoop(Timer.java:512)
> >     at java.util.TimerThread.run(Timer.java:462)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Low Memory Detector" daemon prio=10 tid=0x08235800 nid=0x84f runnable
> > [0x00000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "CompilerThread1" daemon prio=10 tid=0x9f4a9800 nid=0x84e waiting on
> > condition [0x00000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "CompilerThread0" daemon prio=10 tid=0x9f4a7800 nid=0x84d waiting on
> > condition [0x00000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Signal Dispatcher" daemon prio=10 tid=0x9f4a5c00 nid=0x84c runnable
> > [0x00000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Finalizer" daemon prio=10 tid=0x9f497400 nid=0x84b in Object.wait()
> > [0x9f194000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0xa398ce08> (a java.lang.ref.ReferenceQueue$Lock)
> >     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> >     - locked <0xa398ce08> (a java.lang.ref.ReferenceQueue$Lock)
> >     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> >     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "Reference Handler" daemon prio=10 tid=0x9f496000 nid=0x84a in
> > Object.wait() [0x9f1e5000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0xa397b6e8> (a java.lang.ref.Reference$Lock)
> >     at java.lang.Object.wait(Object.java:485)
> >     at java.lang.ref.Reference
> > $ReferenceHandler.run(Reference.java:116)
> >     - locked <0xa397b6e8> (a java.lang.ref.Reference$Lock)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "main" prio=10 tid=0x08224400 nid=0x844 in Object.wait() [0xb6a06000]
> >    java.lang.Thread.State: WAITING (on object monitor)
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0xaf50ac30> (a
> > org.griphyn.vdl.karajan.VDL2ExecutionContext)
> >     at java.lang.Object.wait(Object.java:485)
> >     at
> >
> org.globus.cog.karajan.workflow.ExecutionContext.waitFor(ExecutionContext.java:226)
> >     - locked <0xaf50ac30> (a
> > org.griphyn.vdl.karajan.VDL2ExecutionContext)
> >     at org.griphyn.vdl.karajan.Loader.main(Loader.java:201)
> >
> >    Locked ownable synchronizers:
> >     - None
> >
> > "VM Thread" prio=10 tid=0x9f492400 nid=0x849 runnable
> >
> > "GC task thread#0 (ParallelGC)" prio=10 tid=0x0822b800 nid=0x845
> > runnable
> >
> > "GC task thread#1 (ParallelGC)" prio=10 tid=0x0822cc00 nid=0x846
> > runnable
> >
> > "GC task thread#2 (ParallelGC)" prio=10 tid=0x0822e400 nid=0x847
> > runnable
> >
> > "GC task thread#3 (ParallelGC)" prio=10 tid=0x0822f800 nid=0x848
> > runnable
> >
> > "VM Periodic Task Thread" prio=10 tid=0x9f4b4000 nid=0x850 waiting on
> > condition
> >
> > JNI global references: 1392
> >
> >
> > Found one Java-level deadlock:
> > =============================
> > "pool-1-thread-8":
> >   waiting to lock monitor 0x08b1859c (object 0xaf5ed6c8, a
> > org.griphyn.vdl.karajan.WrapperMap),
> >   which is held by "pool-1-thread-6"
> > "pool-1-thread-6":
> >   waiting to lock monitor 0x9e89178c (object 0xaed9c108, a
> > org.griphyn.vdl.mapping.RootArrayDataNode),
> >   which is held by "pool-1-thread-8"
> >
> > Java stack information for the threads listed above:
> > ===================================================
> > "pool-1-thread-8":
> >     at org.griphyn.vdl.karajan.WrapperMap.close(WrapperMap.java:25)
> >     - waiting to lock <0xaf5ed6c8> (a
> > org.griphyn.vdl.karajan.WrapperMap)
> >     at
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.closeShallow(VDLFunction.java:516)
> >     - locked <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> >
> org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:121)
> >     at
> > org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:49)
> >     - locked <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:71)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
> >     at
> >
> org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227)
> >     at
> > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104)
> >     at
> >
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
> >     at java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:441)
> >     at java.util.concurrent.FutureTask
> > $Sync.innerRun(FutureTask.java:303)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:662)
> > "pool-1-thread-6":
> >     at
> >
> org.griphyn.vdl.mapping.AbstractDataNode.addListener(AbstractDataNode.java:583)
> >     - waiting to lock <0xaed9c108> (a
> > org.griphyn.vdl.mapping.RootArrayDataNode)
> >     at
> >
> org.griphyn.vdl.karajan.DSHandleFutureWrapper.<init>(DSHandleFutureWrapper.java:24)
> >     at
> > org.griphyn.vdl.karajan.WrapperMap.addNodeListener(WrapperMap.java:61)
> >     - locked <0xaf5ed6c8> (a org.griphyn.vdl.karajan.WrapperMap)
> >     at
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.addFutureListener(VDLFunction.java:523)
> >     at org.griphyn.vdl.karajan.lib.Stagein.function(Stagein.java:88)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
> >     at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
> >     at
> >
> org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:227)
> >     at
> > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104)
> >     at
> >
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
> >     at java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:441)
> >     at java.util.concurrent.FutureTask
> > $Sync.innerRun(FutureTask.java:303)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> >     at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:662)
> >
> > Found 1 deadlock.
> >
> >
> > On Sat, Jun 18, 2011 at 2:52 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         Post the entire output of jstack please.
> >
> >
> >         On Sat, 2011-06-18 at 00:08 -0500, Alberto Chavez wrote:
> >         > I already did the svn updates for cog, and swift, rebuilt
> >         swift with
> >         > ant redist, and ant clean + ant dist,
> >         > the test keeps hanging, but I'm missing probably something:
> >         >
> >         >
> >         > $ svn update cog
> >         > At revision 3167.
> >         >
> >         >
> >         > $ cd cog/modules/
> >         > $ svn update swift
> >         > At revision 4632.
> >         >
> >         >
> >         > I did ant redist, and it was successfully built, but the
> >         test is still
> >         > hanging, now it hung on the 11th iteration. However I'm not
> >         quite sure
> >         > if the svn was properly updated, since I did
> >         >
> >         >
> >         > $ jstack -l 11471 | grep addListener
> >         >         at
> >         >
> >
> org.griphyn.vdl.mapping.AbstractDataNode.addListener(AbstractDataNode.java:583)
> >         >         at
> >         >
> >
> org.griphyn.vdl.mapping.AbstractDataNode.addListener(AbstractDataNode.java:583)
> >         >
> >         >
> >         > but Mihael Mentioned that addListener is not
> >         AbstractDataNode in the
> >         > newer version.
> >         > Any thoughts on that?
> >         >
> >         >
> >         > Alberto.
> >         > > Subject: RE: [Swift-devel] Swift unresponsive while using
> >         local
> >         > provider.
> >         > > From: hategan at mcs.anl.gov
> >         > > To: alberto_chavez at live.com
> >         > > CC: ketancmaheshwari at gmail.com;
> >         swift-devel at ci.uchicago.edu
> >         > > Date: Fri, 17 Jun 2011 21:09:04 -0700
> >         > >
> >         > > On Fri, 2011-06-17 at 22:50 -0500, Alberto Chavez wrote:
> >         > > > Oops, Too late.
> >         > > > Do i need to do ant clean , and ant dist again?
> >         > >
> >         > > You're probably fine in most cases with just "ant dist".
> >         But if you
> >         > want
> >         > > to be sure, do what Jonathan is saying: "ant redist"
> >         > >
> >         > >
> >         > >
> >         >
> >
> >
> >
> >
> >         _______________________________________________
> >         Swift-devel mailing list
> >         Swift-devel at ci.uchicago.edu
> >         http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110703/3c5563e7/attachment.html>

From hategan at mcs.anl.gov  Mon Jul  4 00:50:45 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 04 Jul 2011 00:50:45 -0500
Subject: [Swift-devel] Swift unresponsive while using local provider.
In-Reply-To: <CALwLSKoLqnWo5a+nqy-gevFC1F2V7usQVxUZOwNrpMVg_JSQzA@mail.gmail.com>
References: <BAY156-w505A430E3126E1F6F806199C6D0@phx.gbl>
	<1057363198.21380.1308339533392.JavaMail.root@zimbra.anl.gov>
	<BANLkTikEaYj8gJjz_p0OJQCNVpevGMTV5Q@mail.gmail.com>
	<1308340585.14231.0.camel@blabla>
	<BAY156-w65F158DA12C1A99E42124D9C6D0@phx.gbl>
	<1308355792.24489.2.camel@blabla>
	<BAY156-w38F63744968EC2E4ED5ED59C6C0@phx.gbl>
	<1308356386.24582.2.camel@blabla>
	<BAY156-w64DC1F7C60988A1D590FB9C6C0@phx.gbl>
	<1308358465.24760.10.camel@blabla>
	<BAY156-w13B07C2D9777E611BF7BED9C6C0@phx.gbl>
	<1308367911.25750.0.camel@blabla>
	<BAY156-w16A4A1FB055839CE62C7F09C6C0@phx.gbl>
	<1308368849.25915.3.camel@blabla>
	<BAY156-w49B30735AD916251ADB22F9C6C0@phx.gbl>
	<BANLkTim0vb47kjfqwfh65HY1D8ELhi9Hdw@mail.gmail.com>
	<BAY156-w442D26F6D4F07074EB4B649C6C0@phx.gbl>
	<1308370144.26103.1.camel@blabla>
	<BAY156-w8529C910F18014CF13E179C6C0@phx.gbl>
	<1308426730.30334.0.camel@blabla>
	<BANLkTikQ4KaAWhc805=MPKHXhbA_tDPVvQ@mail.gmail.com>
	<1308853713.17296.0.camel@blabla>
	<CALwLSKoLqnWo5a+nqy-gevFC1F2V7usQVxUZOwNrpMVg_JSQzA@mail.gmail.com>
Message-ID: <1309758645.4539.0.camel@blabla>

On Sun, 2011-07-03 at 20:40 -0500, David Kelly wrote:
> This seems to be fixed now with the latest version. The test script
> ran successfully around 2000 times before I killed it.

Thanks for testing it.




From benc at hawaga.org.uk  Mon Jul  4 10:00:25 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 4 Jul 2011 15:00:25 +0000 (GMT)
Subject: [Swift-devel] readData2
In-Reply-To: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
Message-ID: <Pine.LNX.4.64.1107041459090.18458@dildano.hawaga.org.uk>


>     I am going to rename readData2 to something more meaningful.  I am 
> thinking maybe readStructured?  I will keep readData2 as an alias to the 
> renamed function as too not break anyones current code but the userguide 
> will reflect the renamed function.  Any suggestions on the name or is 
> readStructured good?

Something that I did when renaming things before was to make the old name 
issue a warning when used, for a couple of releases, and then remove. That 
was an attempt at a balance between not breaking things and preventing the 
long term build-up of detritus.

-- 



From benc at hawaga.org.uk  Mon Jul  4 11:08:17 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 4 Jul 2011 16:08:17 +0000 (GMT)
Subject: [Swift-devel] finding the execution sites from swift logs
In-Reply-To: <BANLkTik7kQ_gpxL5B3prY-5nam1Bi-+T7g@mail.gmail.com>
References: <BANLkTi=GwLFYvQXJSQN8DHhU5VPjFExi+g@mail.gmail.com>
	<BANLkTimz4HVb+5Yf2153ZDGHaeR4Y6=Lhw@mail.gmail.com>
	<BANLkTim25NCuwgAgAbXwHDpkLTLHkXA-wg@mail.gmail.com>
	<BANLkTikZwEm8+uediG=U-GBBVtF=dLcMJQ@mail.gmail.com>
	<BANLkTinCkoSLgsRm7kEtFhN=6WjZ08T8Qw@mail.gmail.com>
	<BANLkTik7kQ_gpxL5B3prY-5nam1Bi-+T7g@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1107041603000.18458@dildano.hawaga.org.uk>


> Right. the execute2.event target in swift-plot-log gets the time
> duration between JOB_START and JOB_END (correct me if i'm wrong Ben).
>  With that you could get how many jobs were sent to each site.

Not really sure, but I think it consists fo the duration between the 
firstand the last mention of the job in the log file (the log file gets 
converted to execute2.transitions and then tehre is a generic 
transitions->event convertor)

I think execute2.event is the right place to get the raw data for Ketan's 
original question:

> >> >> > Does anyone knows from swift log, how to find how many jobs 
> >> >> > executed on a given site when there is a mix of localhost and 
> >> >> > osg sites?

The colouring scripts assume that column 6 of that file is the site name.

-- 



From hategan at mcs.anl.gov  Tue Jul  5 01:09:41 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 01:09:41 -0500
Subject: [Swift-devel] svn commits
Message-ID: <1309846181.1762.2.camel@blabla>

It seems that our SVN server has run out of disk space.

I would strongly advise against making any commits until this is fixed.
I've already made some that changed the revision number but didn't seem
to have changed actual files, so it seems pretty bad as it is.

I've sent an email to support.

Mihael



From alberto_chavez at live.com  Tue Jul  5 06:10:08 2011
From: alberto_chavez at live.com (Alberto Chavez)
Date: Tue, 5 Jul 2011 06:10:08 -0500
Subject: [Swift-devel] svn commits
In-Reply-To: <1309846181.1762.2.camel@blabla>
References: <1309846181.1762.2.camel@blabla>
Message-ID: <BAY156-w32997569E60BC2D48229099C5F0@phx.gbl>


Mihael, Is that related to the problem that I've been having to add a new directory to language-behaviour folder since last week? I thought it was a tree conflict with svn, but the revision number changes and the whole thing has the latest revision, but the new directory doesn't appear. (The directory is iterators/ ) 
Alberto.

> From: hategan at mcs.anl.gov
> To: swift-devel at ci.uchicago.edu
> Date: Tue, 5 Jul 2011 01:09:41 -0500
> Subject: [Swift-devel] svn commits
> 
> It seems that our SVN server has run out of disk space.
> 
> I would strongly advise against making any commits until this is fixed.
> I've already made some that changed the revision number but didn't seem
> to have changed actual files, so it seems pretty bad as it is.
> 
> I've sent an email to support.
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110705/98d3c526/attachment.html>

From benc at hawaga.org.uk  Tue Jul  5 09:03:36 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 5 Jul 2011 16:03:36 +0200
Subject: [Swift-devel] this feels almost swiftable but not.
Message-ID: <A675396D-467B-4A08-B016-4E3D672B2E97@hawaga.org.uk>

here's something i've been doing with ALCF machines. the workflow feels almost right for swift but not quite. I'm not interested in implementing this myself, really, because I finish working on this project in 6 weeks time, but I thought it would be interesting food for thought.

there's a big physics simulation app that runs on intrepid, the big bluegene. every so often it outputs a dump into its run directory. A dump takes the form of a directory, called runname-<stepnumber>.d/ where stepnumber is a number that increases as the simulation progresses (inside the simulation, it increases by one each step, but dumps are made somewhat irregularly so it goes up with each dump but by a different amount each time).

swift related note: dumps are directories. they contain some files, and the files are named according to the enclosing directory, and the enclosing directory is named according to the name of the job. that's unusual compared to how swift does things.

Once dumps are made, I want to post process them to make plots, in several different ways to generate 2d plots, 3d plots, and (4d!) movies.

Each 2d and 3d plot output comes from a single dump file. The plots are made by invoking the application again, this time on Eureka - the associated viz cluster.

Eureka shares a GPFS file system with intrepid. The dump directories are large: for example, one I just looked at is 184gb.

So: these directories pretty much cannot be copied. They can be symlinked easily enough, though.

The run progresses over many days/weeks. I don't want to wait for "all" the output dumps to be ready before post processing them. Indeed, the definition of "all" is maybe not known programmatically but is decided by looking at the output dumps and deciding (in-head, not in-computer) that we've seen enough.

Swift has almost but not quite enough behaviour at the moment to deal with mapping a data set that increases in size over time like this (rather than being atomic wrt a single swift run), though its been talked about on swift devel. I want to have swift generate the plots for a dump as soon as the dump is made (where "as soon as" might be hours later, given the timescales involved, but not weeks later).

I don't want to regenerate plots that have already been plotted.

Except that sometimes I want to make a new set of plots with different parameters (for example, different viewing angle, plotting different variables, different parameters to viz method).

Sometimes I will want both the older and the newer plots to continue being generated over time. Sometimes I will be done with the older (or the newer) one, and want to stop one of them from being generated any more.

The 3d plot happens as two stage: first the simulation application is invoked on eureka to produce a directory (per dump step) of .silo files. this end up occupying megabytes (so much smaller than the raw dumps but still not trivial to copy around. secondly, a third party application, VisIt, is used to render those silo files into PNGs.
When I talk about generating different 3d plots above - the silo files do not change. The different plots are specified by parameters to this visit step.
That interacts awkwardly with i) silo files being stored as collections of files in directories, where the silo dump is the directory itself; and ii) how ongoing runs happen: a dump (directory) should turn into a silo (directory) only once, soon after the dump itself is created.

There's an additional interesting thing with VisIt: it allocates its own workers, and this startup time is non trivial wrt the cost of rendering a single silo dump. VisIt has a python API and I feed in a list of silo dumps that should be rendered, and these are then all rendered inside a single run of VisIt, resulting in some of the silo dumps turning into output PNGs; and some not because I ran out of wall time. At which point, I manually trim the list and submit again. What's interesting there is that I'm launching VisIt with a specification of things to work on, knowing that the specific visit run will overall fail - because I'm interested in the more granular production of individual plot PNG file. How that interacts with Swift seems interesting too: there's an "unreliable worker" that I feed in multiple tasks to, and those individual tasks may succeed or fail and can be restarted/rerun independent of the worker. There would also be some scope for high level parallelisation. I am using 30 worker nodes per visit run which means I could have 3 sets of these going at any one time on Eureka if it was lightly loaded.

Like I said at the top, I'm not interested in implementing this in Swift, but I think it gives some interesting pokes at the envelope of what can/cannot be done with swift.

Ben



From dsk at ci.uchicago.edu  Tue Jul  5 09:23:42 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Tue, 5 Jul 2011 09:23:42 -0500
Subject: [Swift-devel] this feels almost swiftable but not.
In-Reply-To: <A675396D-467B-4A08-B016-4E3D672B2E97@hawaga.org.uk>
References: <A675396D-467B-4A08-B016-4E3D672B2E97@hawaga.org.uk>
Message-ID: <4539BAC0-8649-4943-9B77-F7C31BF999E9@ci.uchicago.edu>

Just scanning this, one thought (that I'm not sure of): would it be worth a discussion between Swift and VisIt to talk about integration?

Dan


On Jul 5, 2011, at 9:03 AM, Ben Clifford wrote:

> here's something i've been doing with ALCF machines. the workflow feels almost right for swift but not quite. I'm not interested in implementing this myself, really, because I finish working on this project in 6 weeks time, but I thought it would be interesting food for thought.
> 
> there's a big physics simulation app that runs on intrepid, the big bluegene. every so often it outputs a dump into its run directory. A dump takes the form of a directory, called runname-<stepnumber>.d/ where stepnumber is a number that increases as the simulation progresses (inside the simulation, it increases by one each step, but dumps are made somewhat irregularly so it goes up with each dump but by a different amount each time).
> 
> swift related note: dumps are directories. they contain some files, and the files are named according to the enclosing directory, and the enclosing directory is named according to the name of the job. that's unusual compared to how swift does things.
> 
> Once dumps are made, I want to post process them to make plots, in several different ways to generate 2d plots, 3d plots, and (4d!) movies.
> 
> Each 2d and 3d plot output comes from a single dump file. The plots are made by invoking the application again, this time on Eureka - the associated viz cluster.
> 
> Eureka shares a GPFS file system with intrepid. The dump directories are large: for example, one I just looked at is 184gb.
> 
> So: these directories pretty much cannot be copied. They can be symlinked easily enough, though.
> 
> The run progresses over many days/weeks. I don't want to wait for "all" the output dumps to be ready before post processing them. Indeed, the definition of "all" is maybe not known programmatically but is decided by looking at the output dumps and deciding (in-head, not in-computer) that we've seen enough.
> 
> Swift has almost but not quite enough behaviour at the moment to deal with mapping a data set that increases in size over time like this (rather than being atomic wrt a single swift run), though its been talked about on swift devel. I want to have swift generate the plots for a dump as soon as the dump is made (where "as soon as" might be hours later, given the timescales involved, but not weeks later).
> 
> I don't want to regenerate plots that have already been plotted.
> 
> Except that sometimes I want to make a new set of plots with different parameters (for example, different viewing angle, plotting different variables, different parameters to viz method).
> 
> Sometimes I will want both the older and the newer plots to continue being generated over time. Sometimes I will be done with the older (or the newer) one, and want to stop one of them from being generated any more.
> 
> The 3d plot happens as two stage: first the simulation application is invoked on eureka to produce a directory (per dump step) of .silo files. this end up occupying megabytes (so much smaller than the raw dumps but still not trivial to copy around. secondly, a third party application, VisIt, is used to render those silo files into PNGs.
> When I talk about generating different 3d plots above - the silo files do not change. The different plots are specified by parameters to this visit step.
> That interacts awkwardly with i) silo files being stored as collections of files in directories, where the silo dump is the directory itself; and ii) how ongoing runs happen: a dump (directory) should turn into a silo (directory) only once, soon after the dump itself is created.
> 
> There's an additional interesting thing with VisIt: it allocates its own workers, and this startup time is non trivial wrt the cost of rendering a single silo dump. VisIt has a python API and I feed in a list of silo dumps that should be rendered, and these are then all rendered inside a single run of VisIt, resulting in some of the silo dumps turning into output PNGs; and some not because I ran out of wall time. At which point, I manually trim the list and submit again. What's interesting there is that I'm launching VisIt with a specification of things to work on, knowing that the specific visit run will overall fail - because I'm interested in the more granular production of individual plot PNG file. How that interacts with Swift seems interesting too: there's an "unreliable worker" that I feed in multiple tasks to, and those individual tasks may succeed or fail and can be restarted/rerun independent of the worker. There would also be some scope for high level parallelisati
> on. I am using 30 worker nodes per visit run which means I could have 3 sets of these going at any one time on Eureka if it was lightly loaded.
> 
> Like I said at the top, I'm not interested in implementing this in Swift, but I think it gives some interesting pokes at the envelope of what can/cannot be done with swift.
> 
> Ben
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/






From ketancmaheshwari at gmail.com  Tue Jul  5 11:38:46 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Jul 2011 11:38:46 -0500
Subject: [Swift-devel] Hangchecker tweak
In-Reply-To: <1309372335.29757.3.camel@blabla>
References: <BANLkTimDJW-G44_m3Skbg_mHTziY0a9vJw@mail.gmail.com>
	<1309366102.28552.1.camel@blabla>
	<BANLkTi=XoT1A4BfueO1bK7dbXLUzu06x8A@mail.gmail.com>
	<1309372335.29757.3.camel@blabla>
Message-ID: <CAMUuvirbGU441w-KFYoBCQKPz+pqjKrpqXpx8sbp1Rx3-46eFg@mail.gmail.com>

On Wed, Jun 29, 2011 at 1:32 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> I strongly suspect that the hangs are not due to the hang checker.
>

This may be right. After many experiments (about 20) with large scale (upto
60-slots, 4-node) submissions with trunk, it seems that the jobs just do not
get submitted after a low arbitrary submissions.

Things that I observe with trunk on Beagle:

1. Disproportionate number of stage-ins happen when compared to the intended
number of jobs: for a 10-slot 4-node setup, 4980 stage-ins

2. The submit file created contained "node=" lined for 4-node jobs and not
for 2-node ones. I changed the use.mppwidth=false entry in
provider-pbs.properties to true. However, I do not know why this was
happening for 4-node jobs and not for the 2-node ones.

3. I see intermittent write failures from pbs to the swift.workdir with
"failed to transfer wrapper log messages".

Debugging more.

Ketan



> On Wed, 2011-06-29 at 13:02 -0500, Ketan Maheshwari wrote:
> >
> > I built Swift with this change and submitted a run with throttle value
> > of 3600 app tasks. It seems to be working. I see 3600 PBS jobs have
> > been submitted to Beagle.
> >
> >
> > On Wed, Jun 29, 2011 at 11:48 AM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         On Wed, 2011-06-29 at 11:36 -0500, Ketan Maheshwari wrote:
> >
> >         > To confirm the hypothesis, could you indicate how could I
> >         disable the
> >         > hangchecker or increase the time period before it gets
> >         invoked.
> >
> >
> >         in Loader.main(), comment out the 'new
> >         HangChecker(stack).start()' line.
> >
> >
> >
> >
> > --
> > Ketan
> >
> >
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110705/4cad8e18/attachment.html>

From hategan at mcs.anl.gov  Tue Jul  5 12:03:25 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 12:03:25 -0500
Subject: [Swift-devel] svn commits
In-Reply-To: <1309846181.1762.2.camel@blabla>
References: <1309846181.1762.2.camel@blabla>
Message-ID: <1309885405.4432.0.camel@blabla>

Seems to be fixed. And it seems like commits made during that period
eventually made it through, so it's not as bad as I thought it is.

Mihael

On Tue, 2011-07-05 at 01:09 -0500, Mihael Hategan wrote:
> It seems that our SVN server has run out of disk space.
> 
> I would strongly advise against making any commits until this is fixed.
> I've already made some that changed the revision number but didn't seem
> to have changed actual files, so it seems pretty bad as it is.
> 
> I've sent an email to support.
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Tue Jul  5 12:04:59 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 12:04:59 -0500
Subject: [Swift-devel] svn commits
In-Reply-To: <BAY156-w32997569E60BC2D48229099C5F0@phx.gbl>
References: <1309846181.1762.2.camel@blabla>
	<BAY156-w32997569E60BC2D48229099C5F0@phx.gbl>
Message-ID: <1309885499.4432.2.camel@blabla>

On Tue, 2011-07-05 at 06:10 -0500, Alberto Chavez wrote:
> Mihael, 
> Is that related to the problem that I've been having to add a new
> directory to language-behaviour folder since last week? 
> I thought it was a tree conflict with svn, but the revision number
> changes and the whole thing has the latest revision, but the new
> directory doesn't appear. (The directory is iterators/ ) 

I'm unsure. Can you do an "svn stat" in language-behaviour and post the
output?






From hategan at mcs.anl.gov  Tue Jul  5 12:13:04 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 12:13:04 -0500
Subject: [Swift-devel] branch
Message-ID: <1309885984.4432.3.camel@blabla>

I branched:
cog: branches/4.1.9
swift: branches/release-0.93



From hategan at mcs.anl.gov  Tue Jul  5 12:15:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 12:15:34 -0500
Subject: [Swift-devel] provenance
Message-ID: <1309886134.4432.6.camel@blabla>

In trunk provenance inside the compiled file is disabled by default. It
is my understanding that it was more common for it to not be used than
otherwise, but if this isn't so, let me know.

It can be enabled with "-enable.provenance" on the swift command line.





From hategan at mcs.anl.gov  Tue Jul  5 13:08:42 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 13:08:42 -0500
Subject: [Swift-devel] uninitialized variables...
Message-ID: <1309889322.4953.1.camel@blabla>

... there's a patch committed now to detect them at compile time.

Previously the following would hang:

int a, b;

int c = a + b;

trace(c);

It now fails at compile time with a complaint that a is uninitialized.



From hategan at mcs.anl.gov  Tue Jul  5 13:39:39 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 13:39:39 -0500
Subject: [Swift-devel] iterate behavior
Message-ID: <1309891179.4953.4.camel@blabla>

I have

iterate i {
  trace(i);
} until (i > 2);

I read that "until" as "the iteration body should not be run if i > 2",
and yet trace goes up to 3. And I'm not sure if that's the right way to
go.

Discuss...



From dsk at ci.uchicago.edu  Tue Jul  5 13:46:12 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Tue, 5 Jul 2011 13:46:12 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <1309891179.4953.4.camel@blabla>
References: <1309891179.4953.4.camel@blabla>
Message-ID: <6FEB5CBE-9786-4F3E-BA45-524B29CCDD4F@ci.uchicago.edu>

This seems right to me.  When i = 2, the until is false so the next iteration takes place.  Then i = 3 and the until is true, so the structure stops.

On the other hand,
iterate i {
  trace(i);
} while (i <= 2);

would not have a trace at 3, in my opinion...

Dan


On Jul 5, 2011, at 1:39 PM, Mihael Hategan wrote:

> I have
> 
> iterate i {
>  trace(i);
> } until (i > 2);
> 
> I read that "until" as "the iteration body should not be run if i > 2",
> and yet trace goes up to 3. And I'm not sure if that's the right way to
> go.
> 
> Discuss...
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/






From benc at hawaga.org.uk  Tue Jul  5 13:50:26 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 5 Jul 2011 18:50:26 +0000 (GMT)
Subject: [Swift-devel] iterate behavior
In-Reply-To: <1309891179.4953.4.camel@blabla>
References: <1309891179.4953.4.camel@blabla>
Message-ID: <Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>


> iterate i {
>   trace(i);
> } until (i > 2);
> 
> I read that "until" as "the iteration body should not be run if i > 2",
> and yet trace goes up to 3. And I'm not sure if that's the right way to
> go.
> 
> Discuss...

closest C-like equivalent (and thus something to emulate) is:

main() {

 int i=0;
 do {
   printf("%d\n",i);
   i++;
 } while(!(i>2));
}

which says:

$ ./a.out 
0
1
2

I suspect when I implemented that I was thinking of the i++ being after 
the condition check, rather than before, but consistency suggests that I 
was wrong.

-- 


From jonmon at utexas.edu  Tue Jul  5 13:53:08 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Tue, 5 Jul 2011 13:53:08 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>
References: <1309891179.4953.4.camel@blabla>
	<Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>
Message-ID: <459B1CFA-D0D2-4C56-BFFF-4FFBBDE93936@utexas.edu>

I do not think it should be like that.  I understand why it would do that.  The java code for that would be:

int i = 0;
do
{
    System.out.println( i );
} while( i++ <= 2 );

but in my opinion the increment of i should be done before the check.  I think stopping at 2 is more intuitive.  In terms of Java:

int i = 0;
do
{
    System.out.println( i );
} while( ++i <= 2 );

On Jul 5, 2011, at 1:50 PM, Ben Clifford wrote:

> 
>> iterate i {
>>  trace(i);
>> } until (i > 2);
>> 
>> I read that "until" as "the iteration body should not be run if i > 2",
>> and yet trace goes up to 3. And I'm not sure if that's the right way to
>> go.
>> 
>> Discuss...
> 
> closest C-like equivalent (and thus something to emulate) is:
> 
> main() {
> 
> int i=0;
> do {
>   printf("%d\n",i);
>   i++;
> } while(!(i>2));
> }
> 
> which says:
> 
> $ ./a.out 
> 0
> 1
> 2
> 
> I suspect when I implemented that I was thinking of the i++ being after 
> the condition check, rather than before, but consistency suggests that I 
> was wrong.
> 
> -- 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From ketancmaheshwari at gmail.com  Tue Jul  5 14:04:11 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Jul 2011 14:04:11 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <1309891179.4953.4.camel@blabla>
References: <1309891179.4953.4.camel@blabla>
Message-ID: <CAMUuviotMPbo72Ty-hWKOTMsaFgKogW1shr5MTkcMfn38LGqjQ@mail.gmail.com>

The first instance 'until' i gets greater than 2 is when it gets 3. With
this regard, I think trace could go ahead and show the value of i but
somehow no other statement should get evaluated in that iteration. This is
justified imho since if one does manual tracing, it would go something like
this:

i=0
i>2? No
---
i=1
i>2? No
---
i=2
i>2? No
---
i=3
i>2? Yes
---

On Tue, Jul 5, 2011 at 1:39 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> I have
>
> iterate i {
>  trace(i);
> } until (i > 2);
>
> I read that "until" as "the iteration body should not be run if i > 2",
> and yet trace goes up to 3. And I'm not sure if that's the right way to
> go.
>
> Discuss...
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110705/1ad955c2/attachment.html>

From benc at hawaga.org.uk  Tue Jul  5 14:09:38 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 5 Jul 2011 19:09:38 +0000 (GMT)
Subject: [Swift-devel] iterate behavior
In-Reply-To: <CAMUuviotMPbo72Ty-hWKOTMsaFgKogW1shr5MTkcMfn38LGqjQ@mail.gmail.com>
References: <1309891179.4953.4.camel@blabla>
	<CAMUuviotMPbo72Ty-hWKOTMsaFgKogW1shr5MTkcMfn38LGqjQ@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1107051905100.18458@dildano.hawaga.org.uk>


> The first instance 'until' i gets greater than 2 is when it gets 3. With
> this regard, I think trace could go ahead and show the value of i but
> somehow no other statement should get evaluated in that iteration.

I think that's crazy: either the body gets executed or it doesn't get 
executed. trace should not be a 'special statement' that enclosing control 
structures need to know about that is distinct from a regular swift 
function call. its also crazy to have trace showing that the body was 
entered and executed when it wasn't. "I can't see whats going wrong: trace 
tells me that this loop iteration happens, but its as if the rest of the 
body doesn't happen..."

-- 


From hategan at mcs.anl.gov  Tue Jul  5 14:12:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 14:12:34 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <459B1CFA-D0D2-4C56-BFFF-4FFBBDE93936@utexas.edu>
References: <1309891179.4953.4.camel@blabla>
	<Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>
	<459B1CFA-D0D2-4C56-BFFF-4FFBBDE93936@utexas.edu>
Message-ID: <1309893154.4953.6.camel@blabla>

On Tue, 2011-07-05 at 13:53 -0500, Jonathan Monette wrote:
> I do not think it should be like that.  I understand why it would do that.  The java code for that would be:
> 
> int i = 0;
> do
> {
>     System.out.println( i );
> } while( i++ <= 2 );
> 
> but in my opinion the increment of i should be done before the check.

Smells right to me. I think that the while/until at the end suggests
that the test is the latest thing to happen in an iteration (i.e. after
the increment).




From jonmon at utexas.edu  Tue Jul  5 14:15:26 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Tue, 5 Jul 2011 14:15:26 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <1309893154.4953.6.camel@blabla>
References: <1309891179.4953.4.camel@blabla>
	<Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>
	<459B1CFA-D0D2-4C56-BFFF-4FFBBDE93936@utexas.edu>
	<1309893154.4953.6.camel@blabla>
Message-ID: <B7259666-0003-4367-A941-F3B9213C008D@utexas.edu>

that is how I would read the iterate statement.

On Jul 5, 2011, at 2:12 PM, Mihael Hategan wrote:

> On Tue, 2011-07-05 at 13:53 -0500, Jonathan Monette wrote:
>> I do not think it should be like that.  I understand why it would do that.  The java code for that would be:
>> 
>> int i = 0;
>> do
>> {
>>    System.out.println( i );
>> } while( i++ <= 2 );
>> 
>> but in my opinion the increment of i should be done before the check.
> 
> Smells right to me. I think that the while/until at the end suggests
> that the test is the latest thing to happen in an iteration (i.e. after
> the increment).
> 
> 



From ketancmaheshwari at gmail.com  Tue Jul  5 14:25:07 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Jul 2011 14:25:07 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <Pine.LNX.4.64.1107051905100.18458@dildano.hawaga.org.uk>
References: <1309891179.4953.4.camel@blabla>
	<CAMUuviotMPbo72Ty-hWKOTMsaFgKogW1shr5MTkcMfn38LGqjQ@mail.gmail.com>
	<Pine.LNX.4.64.1107051905100.18458@dildano.hawaga.org.uk>
Message-ID: <CAMUuvirr1tMmmV5+6J0ubrNXWbV4p82+aPTHLQd3pXDy2f-AOQ@mail.gmail.com>

On Tue, Jul 5, 2011 at 2:09 PM, Ben Clifford <benc at hawaga.org.uk> wrote:

>
> > The first instance 'until' i gets greater than 2 is when it gets 3. With
> > this regard, I think trace could go ahead and show the value of i but
> > somehow no other statement should get evaluated in that iteration.
>
> I think that's crazy: either the body gets executed or it doesn't get
> executed. trace should not be a 'special statement' that enclosing control
> structures need to know about that is distinct from a regular swift
> function call. its also crazy to have trace showing that the body was
> entered and executed when it wasn't. "I can't see whats going wrong: trace
> tells me that this loop iteration happens, but its as if the rest of the
> body doesn't happen..."
>

trace could be made a debug-level statement and can represent a "visual"
debugging tool in a script-based programming paradigm.


>
> --
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110705/6dbf1708/attachment.html>

From ketancmaheshwari at gmail.com  Tue Jul  5 15:19:53 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 5 Jul 2011 15:19:53 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <CAMUuvirr1tMmmV5+6J0ubrNXWbV4p82+aPTHLQd3pXDy2f-AOQ@mail.gmail.com>
References: <1309891179.4953.4.camel@blabla>
	<CAMUuviotMPbo72Ty-hWKOTMsaFgKogW1shr5MTkcMfn38LGqjQ@mail.gmail.com>
	<Pine.LNX.4.64.1107051905100.18458@dildano.hawaga.org.uk>
	<CAMUuvirr1tMmmV5+6J0ubrNXWbV4p82+aPTHLQd3pXDy2f-AOQ@mail.gmail.com>
Message-ID: <CAMUuvipHdUR6aLc=bqTYct+q22oB91rRjJ8_THVPZocDZcyYvg@mail.gmail.com>

On Tue, Jul 5, 2011 at 2:25 PM, Ketan Maheshwari <ketancmaheshwari at gmail.com
> wrote:

>
>
> On Tue, Jul 5, 2011 at 2:09 PM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
>>
>> > The first instance 'until' i gets greater than 2 is when it gets 3. With
>> > this regard, I think trace could go ahead and show the value of i but
>> > somehow no other statement should get evaluated in that iteration.
>>
>> I think that's crazy: either the body gets executed or it doesn't get
>> executed. trace should not be a 'special statement' that enclosing control
>> structures need to know about that is distinct from a regular swift
>> function call. its also crazy to have trace showing that the body was
>> entered and executed when it wasn't. "I can't see whats going wrong: trace
>> tells me that this loop iteration happens, but its as if the rest of the
>> body doesn't happen..."
>>
>
> trace could be made a debug-level statement and can represent a "visual"
> debugging tool in a script-based programming paradigm.
>

Furthermore, a debug-level trace could be useful applied to arrays:

foreach anelement, idx in anarray{
res[ix] = some_func(anelement);
trace(anarray);
}

Making trace(anarray) print the "end-of-array:anarray" without really
executing some_func could be a useful debugging tool.


>
>
>>
>> --
>>
>
>
>
> --
> Ketan
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110705/7ab1a859/attachment.html>

From hategan at mcs.anl.gov  Tue Jul  5 15:43:04 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 15:43:04 -0500
Subject: [Swift-devel] iterate behavior
In-Reply-To: <B7259666-0003-4367-A941-F3B9213C008D@utexas.edu>
References: <1309891179.4953.4.camel@blabla>
	<Pine.LNX.4.64.1107051847140.18458@dildano.hawaga.org.uk>
	<459B1CFA-D0D2-4C56-BFFF-4FFBBDE93936@utexas.edu>
	<1309893154.4953.6.camel@blabla>
	<B7259666-0003-4367-A941-F3B9213C008D@utexas.edu>
Message-ID: <1309898584.14907.0.camel@blabla>

Ok. This is how it works in trunk now.

On Tue, 2011-07-05 at 14:15 -0500, Jonathan Monette wrote:
> that is how I would read the iterate statement.
> 
> On Jul 5, 2011, at 2:12 PM, Mihael Hategan wrote:
> 
> > On Tue, 2011-07-05 at 13:53 -0500, Jonathan Monette wrote:
> >> I do not think it should be like that.  I understand why it would do that.  The java code for that would be:
> >> 
> >> int i = 0;
> >> do
> >> {
> >>    System.out.println( i );
> >> } while( i++ <= 2 );
> >> 
> >> but in my opinion the increment of i should be done before the check.
> > 
> > Smells right to me. I think that the while/until at the end suggests
> > that the test is the latest thing to happen in an iteration (i.e. after
> > the increment).
> > 
> > 
> 




From hategan at mcs.anl.gov  Tue Jul  5 16:51:41 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 05 Jul 2011 16:51:41 -0500
Subject: [Swift-devel] variable cleanup
Message-ID: <1309902701.27715.5.camel@blabla>

There is code in trunk now to clean up variables when they go out of
scope. This applies to compound procedure scopes, iterate scopes,
foreach scopes and if/[then|else]. And this reminds me, the main/program
scope should also have this.

Cleanup means that:
- some of the in-memory data is nullified. I'm not sure this is
necessary since the data should be garbage collected by the java GC.
- some mappers (in particular ones advertising themselves as temporary -
e.g. concurrent) should remove the files that they created

Sadly this isn't fully what I wanted, for example:

file a[] <some proper mapper>;

foreach i in [1:1000000] {
  a[i] = generate(i);
}

In the above case it would be nice if no memory reference would be held
to the various a[i] after they are "assigned".



From wilde at mcs.anl.gov  Tue Jul  5 19:02:48 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 5 Jul 2011 19:02:48 -0500 (CDT)
Subject: [Swift-devel] variable cleanup
In-Reply-To: <1309902701.27715.5.camel@blabla>
Message-ID: <1680847804.68094.1309910568266.JavaMail.root@zimbra.anl.gov>

Great!  Allan, can you see if this now meets the NCAR needs for removal of temporary files?

- Mike


----- Original Message -----
> There is code in trunk now to clean up variables when they go out of
> scope. This applies to compound procedure scopes, iterate scopes,
> foreach scopes and if/[then|else]. And this reminds me, the
> main/program
> scope should also have this.
> 
> Cleanup means that:
> - some of the in-memory data is nullified. I'm not sure this is
> necessary since the data should be garbage collected by the java GC.
> - some mappers (in particular ones advertising themselves as temporary
> -
> e.g. concurrent) should remove the files that they created
> 
> Sadly this isn't fully what I wanted, for example:
> 
> file a[] <some proper mapper>;
> 
> foreach i in [1:1000000] {
> a[i] = generate(i);
> }
> 
> In the above case it would be nice if no memory reference would be
> held
> to the various a[i] after they are "assigned".
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From davidkelly999 at gmail.com  Wed Jul  6 11:16:38 2011
From: davidkelly999 at gmail.com (David Kelly)
Date: Wed, 6 Jul 2011 11:16:38 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <alpine.WNT.2.00.1107012055280.912@justinwozniak>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
Message-ID: <CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>

Would it make sense to replace readData and readStructured with one flexible
function which could use multiple formats? For example:

foo = read( -type="JSON", -file="myfile.json" );

I think JSON, XML, and CSV would be useful to have in addition to the two
existing Swift formats.

David

On Fri, Jul 1, 2011 at 9:04 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

>
> fopen(3) is smart enough to know that "r" means read.  Certainly Swift is
> smarter than fopen.
>
> However, I think there is a lot of value in allowing users to read
> unformatted, "one value per line" data and I don't think we should force
> users to convert possibly large text files into Swift format.  So, I think
> there is value in keeping the two functionalities separate.
>
> If we're going to mandate a file format, we should try to be compatible
> with JSON or something.  But I don't think we should mandate a file
> format.
>
>        Justin
>
> On Fri, 1 Jul 2011, Tim Armstrong wrote:
>
> > You could go full unix shell style and call it "rd"
> >
> > On Fri, Jul 1, 2011 at 6:09 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> >
> >> On Fri, 2011-07-01 at 18:04 -0700, Sarah Kenny wrote:
> >>> my 2 cents...readStruct...slightly more succinct :)
> >>
> >> 2 more cents: read. Can't get much better than that.
> >>>
> >>> On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette <jonmon at utexas.edu>
> >>> wrote:
> >>>         I am all for dropping readData. writeData would have to be
> >>>         modified to output data for "readData2".
> >>>
> >>>         Do you have a better name? I thought maybe readMapping or
> >>>         readExplicitMapping but Justin said the mapping term maybe
> >>>         should not be used in this case.
> >>>
> >>>
> >>>         On Jul 1, 2011 6:12 PM, "Mihael Hategan" <hategan at mcs.anl.gov>
> >>>         wrote:
> >>>        > On Thu, 2011-06-30 at 13:40 -0500, Jonathan Monette wrote:
> >>>        >> Hello,
> >>>        >> I am going to rename readData2 to something more
> >>>         meaningful. I am
> >>>        >> thinking maybe readStructured? I will keep readData2 as an
> >>>         alias to
> >>>        >> the renamed function as too not break anyones current code
> >>>         but the
> >>>        >> userguide will reflect the renamed function. Any
> >>>         suggestions on the
> >>>        >> name or is readStructured good?
> >>>        >
> >>>        > I think we should drop readData and slowly replace it with
> >>>         readData2.
> >>>        > readStructured doesn't seem to be more illuminating to me.
> >>>        >
> >>>        >
> >>>
> >>>
> >>>         _______________________________________________
> >>>         Swift-devel mailing list
> >>>         Swift-devel at ci.uchicago.edu
> >>>
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Sarah Kenny
> >>> Programmer
> >>> University of Chicago, Computation Institute
> >>> University of California Irvine, Dept. of Neurology
> >>> 773-818-8300
> >>>
> >>
> >>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>
> >
>
> --
> Justin M Wozniak
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110706/3f07cd3a/attachment.html>

From hategan at mcs.anl.gov  Wed Jul  6 12:30:26 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 06 Jul 2011 12:30:26 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
Message-ID: <1309973426.28880.4.camel@blabla>

I think that the first priority is to have one format able to support
all swift structures. And that is what readData2 did. But I like the
idea of being able to supply a format as you mention below.

So it's probably a smart thing to not rename anything at this point but
introduce the read(format=) scheme below and slowly deprecate readData
and readData2.

On Wed, 2011-07-06 at 11:16 -0500, David Kelly wrote:
> Would it make sense to replace readData and readStructured with one
> flexible function which could use multiple formats? For example: 
> 
> foo = read( -type="JSON", -file="myfile.json" );
> 
> I think JSON, XML, and CSV would be useful to have in addition to the
> two existing Swift formats.
> 
> David
> 
> On Fri, Jul 1, 2011 at 9:04 PM, Justin M Wozniak <wozniak at mcs.anl.gov>
> wrote:
>         
>         fopen(3) is smart enough to know that "r" means read.
>          Certainly Swift is
>         smarter than fopen.
>         
>         However, I think there is a lot of value in allowing users to
>         read
>         unformatted, "one value per line" data and I don't think we
>         should force
>         users to convert possibly large text files into Swift format.
>          So, I think
>         there is value in keeping the two functionalities separate.
>         
>         If we're going to mandate a file format, we should try to be
>         compatible
>         with JSON or something.  But I don't think we should mandate a
>         file
>         format.
>         
>                Justin
>         
>         
>         On Fri, 1 Jul 2011, Tim Armstrong wrote:
>         
>         > You could go full unix shell style and call it "rd"
>         >
>         > On Fri, Jul 1, 2011 at 6:09 PM, Mihael Hategan
>         <hategan at mcs.anl.gov> wrote:
>         >
>         >> On Fri, 2011-07-01 at 18:04 -0700, Sarah Kenny wrote:
>         >>> my 2 cents...readStruct...slightly more succinct :)
>         >>
>         >> 2 more cents: read. Can't get much better than that.
>         >>>
>         >>> On Fri, Jul 1, 2011 at 4:22 PM, Jonathan S Monette
>         <jonmon at utexas.edu>
>         >>> wrote:
>         >>>         I am all for dropping readData. writeData would
>         have to be
>         >>>         modified to output data for "readData2".
>         >>>
>         >>>         Do you have a better name? I thought maybe
>         readMapping or
>         >>>         readExplicitMapping but Justin said the mapping
>         term maybe
>         >>>         should not be used in this case.
>         >>>
>         >>>
>         >>>         On Jul 1, 2011 6:12 PM, "Mihael Hategan"
>         <hategan at mcs.anl.gov>
>         >>>         wrote:
>         >>>        > On Thu, 2011-06-30 at 13:40 -0500, Jonathan
>         Monette wrote:
>         >>>        >> Hello,
>         >>>        >> I am going to rename readData2 to something more
>         >>>         meaningful. I am
>         >>>        >> thinking maybe readStructured? I will keep
>         readData2 as an
>         >>>         alias to
>         >>>        >> the renamed function as too not break anyones
>         current code
>         >>>         but the
>         >>>        >> userguide will reflect the renamed function. Any
>         >>>         suggestions on the
>         >>>        >> name or is readStructured good?
>         >>>        >
>         >>>        > I think we should drop readData and slowly
>         replace it with
>         >>>         readData2.
>         >>>        > readStructured doesn't seem to be more
>         illuminating to me.
>         >>>        >
>         >>>        >
>         >>>
>         >>>
>         >>>         _______________________________________________
>         >>>         Swift-devel mailing list
>         >>>         Swift-devel at ci.uchicago.edu
>         >>>
>         >>
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         >>>
>         >>>
>         >>>
>         >>>
>         >>> --
>         >>> Sarah Kenny
>         >>> Programmer
>         >>> University of Chicago, Computation Institute
>         >>> University of California Irvine, Dept. of Neurology
>         >>> 773-818-8300
>         >>>
>         >>
>         >>
>         >> _______________________________________________
>         >> Swift-devel mailing list
>         >> Swift-devel at ci.uchicago.edu
>         >>
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         >>
>         >
>         
>         
>         --
>         Justin M Wozniak
>         
>         _______________________________________________
>         Swift-devel mailing list
>         Swift-devel at ci.uchicago.edu
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From benc at hawaga.org.uk  Wed Jul  6 12:41:45 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 6 Jul 2011 17:41:45 +0000 (GMT)
Subject: [Swift-devel] readData2
In-Reply-To: <CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>


> Would it make sense to replace readData and readStructured with one flexible
> function which could use multiple formats? For example:
> 
> foo = read( -type="JSON", -file="myfile.json" );

Not that I particularly advocate doing things this way, but here's a 
different approach.

Swift already has a mechanism for specifying the type of a file. Its the 
swift type system.

  CSVFile f <"myfile">;
  foo = read(f);

That doesn't work so well if you want to specify lots of parameters to 
influence the reading, on the read line. But it does hark back to the 
beginning of swift where there was the idea of mapping data into XML.

Potentially you could define the parameterisation in the type, rather than 
in the read() call.

-- 


From hategan at mcs.anl.gov  Wed Jul  6 13:19:40 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 06 Jul 2011 13:19:40 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
	<Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
Message-ID: <1309976380.28880.9.camel@blabla>

On Wed, 2011-07-06 at 17:41 +0000, Ben Clifford wrote:
> > Would it make sense to replace readData and readStructured with one flexible
> > function which could use multiple formats? For example:
> > 
> > foo = read( -type="JSON", -file="myfile.json" );
> 
> Not that I particularly advocate doing things this way, but here's a 
> different approach.
> 
> Swift already has a mechanism for specifying the type of a file. Its the 
> swift type system.
> 
>   CSVFile f <"myfile">;
>   foo = read(f);
> 
> That doesn't work so well if you want to specify lots of parameters to 
> influence the reading, on the read line. But it does hark back to the 
> beginning of swift where there was the idea of mapping data into XML.
> 
> Potentially you could define the parameterisation in the type, rather than 
> in the read() call.

Noope! The structure of a given type is fixed. We want the ability to
read arbitrary such structures/types from a given format. Also for a
given type, there can be many formats. So the format is orthogonal to
the type.




From benc at hawaga.org.uk  Wed Jul  6 13:28:55 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 6 Jul 2011 18:28:55 +0000 (GMT)
Subject: [Swift-devel] readData2
In-Reply-To: <1309976380.28880.9.camel@blabla>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu> 
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com> 
	<alpine.WNT.2.00.1107012055280.912@justinwozniak> 
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com> 
	<Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
	<1309976380.28880.9.camel@blabla>
Message-ID: <Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>



> > Swift already has a mechanism for specifying the type of a file. Its the 
> > swift type system.
> > 
> >   CSVFile f <"myfile">;
> >   foo = read(f);

> Noope! The structure of a given type is fixed. We want the ability to
> read arbitrary such structures/types from a given format. Also for a
> given type, there can be many formats. So the format is orthogonal to
> the type.

CSVFile is a type that indicates that its file is "a csv file". Its not 
the type of the return value (foo above), which is where the variable 
structure occurs. Whether the file content matches the type of foo, beyond 
being a CSV, is outside of the scope of the "this is a CSV file". That's 
the same, I think, whether you use a mapped CSVFile, or whether you use a 
string: its runtime-read-and-casting a loosely typed thing  (CSVFile or 
string) into a specific data structure.

That reminds me of yet another way that was tossed about once: that the 
mapping between file content and in-memory values should be done by a 
mapper.

So rather than:

int[] foo;
foo = read("myfile", ...):

to read  2,3,5,7,11

you instead write:

int[] foo <csvmapper;fn="myfile",...>

-- 


From jonmon at utexas.edu  Wed Jul  6 13:33:46 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Wed, 6 Jul 2011 13:33:46 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
	<Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
	<1309976380.28880.9.camel@blabla>
	<Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>
Message-ID: <20670978-894C-4F03-B6B1-C2609DB8BDCD@utexas.edu>

There is a csv_mapper that maps from a csv file.  I have found that is does not map out all types(correct me if I am wrong and show me how to do this). I believe it assumed everything in the csv file could be mapped to a file not simply just reading data which is what readData and readData2 do.

On Jul 6, 2011, at 1:28 PM, Ben Clifford wrote:

> nt[] foo <csvmapper;fn="myfile",...>



From benc at hawaga.org.uk  Wed Jul  6 13:37:22 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 6 Jul 2011 18:37:22 +0000 (GMT)
Subject: [Swift-devel] readData2
In-Reply-To: <20670978-894C-4F03-B6B1-C2609DB8BDCD@utexas.edu>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu> 
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com> 
	<alpine.WNT.2.00.1107012055280.912@justinwozniak> 
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com> 
	<Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
	<1309976380.28880.9.camel@blabla>
	<Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>
	<20670978-894C-4F03-B6B1-C2609DB8BDCD@utexas.edu>
Message-ID: <Pine.LNX.4.64.1107061836150.17501@dildano.hawaga.org.uk>


> There is a csv_mapper that maps from a csv file.  I have found that is 
> does not map out all types(correct me if I am wrong and show me how to 
> do this). I believe it assumed everything in the csv file could be 
> mapped to a file not simply just reading data which is what readData and 
> readData2 do.

Right. It only maps files. At least in the model in my head, it should be 
able to do values too, but it doesn't.

-- 


From hategan at mcs.anl.gov  Wed Jul  6 14:24:27 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 06 Jul 2011 14:24:27 -0500
Subject: [Swift-devel] readData2
In-Reply-To: <Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>
References: <6BE5DDCC-CD3D-482E-B258-01CCC0E0E64E@utexas.edu>
	<1309561951.30533.1.camel@blabla>
	<BANLkTimL0fKdrkgs-JAJZhzjzCUUd=5Yvw@mail.gmail.com>
	<BANLkTimkrCswSg4WD7Ankti5OppLwQz_ag@mail.gmail.com>
	<1309568984.3651.0.camel@blabla>
	<BANLkTinwffyaS6pORKupjNLqyGaYmcC+9w3cdgaNB8K1mHnkew@mail.gmail.com>
	<alpine.WNT.2.00.1107012055280.912@justinwozniak>
	<CALwLSKpOUfKd6pfBW6Wm_xX1YxP+gah7b+tQ5TMObvNZKud=TQ@mail.gmail.com>
	<Pine.LNX.4.64.1107061737580.18458@dildano.hawaga.org.uk>
	<1309976380.28880.9.camel@blabla>
	<Pine.LNX.4.64.1107061823560.17501@dildano.hawaga.org.uk>
Message-ID: <1309980267.31232.5.camel@blabla>

On Wed, 2011-07-06 at 18:28 +0000, Ben Clifford wrote:
> 
> > > Swift already has a mechanism for specifying the type of a file. Its the 
> > > swift type system.
> > > 
> > >   CSVFile f <"myfile">;
> > >   foo = read(f);
> 
> > Noope! The structure of a given type is fixed. We want the ability to
> > read arbitrary such structures/types from a given format. Also for a
> > given type, there can be many formats. So the format is orthogonal to
> > the type.
> 
> CSVFile is a type that indicates that its file is "a csv file". Its not 
> the type of the return value (foo above), which is where the variable 
> structure occurs. Whether the file content matches the type of foo, beyond 
> being a CSV, is outside of the scope of the "this is a CSV file". That's 
> the same, I think, whether you use a mapped CSVFile, or whether you use a 
> string: its runtime-read-and-casting a loosely typed thing  (CSVFile or 
> string) into a specific data structure.

Ok, better. Sorry I didn't read the first email more carefully.

There's still some cognitive dissonance in my head though.

> 
> That reminds me of yet another way that was tossed about once: that the 
> mapping between file content and in-memory values should be done by a 
> mapper.

Right. A combination between mappers and read/writeData. Essentially
mappers were, in theory, more general, being able to use, for example, a
database. So their implementation was supposed to contain everything
necessary to make the actual storage transparent (and that included
reading actual data). readData is not portable outside files, or at
least not implicitly.



From wozniak at mcs.anl.gov  Wed Jul  6 14:36:26 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 6 Jul 2011 14:36:26 -0500 (CDT)
Subject: [Swift-devel] Call today
Message-ID: <alpine.DEB.2.00.1107061435540.1980@wozniak-desktop.mcs.anl.gov>


I'll be on for a call today to touch base on the latest developments...

-- 
Justin M Wozniak


From skenny at uchicago.edu  Wed Jul  6 15:09:48 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Wed, 6 Jul 2011 13:09:48 -0700
Subject: [Swift-devel] Call today
In-Reply-To: <alpine.DEB.2.00.1107061435540.1980@wozniak-desktop.mcs.anl.gov>
References: <alpine.DEB.2.00.1107061435540.1980@wozniak-desktop.mcs.anl.gov>
Message-ID: <CAE50RW22j+OM-SCFnXYmP8M=28oRrFQ=WnRmvUDyz2vb5KtS1A@mail.gmail.com>

i've been trying to expand the job attributes list that you started, justin:
https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1

feel free to correct/add anything or point me to code :)

On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

>
> I'll be on for a call today to touch base on the latest developments...
>
> --
> Justin M Wozniak
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110706/c4e2c937/attachment.html>

From hategan at mcs.anl.gov  Wed Jul  6 15:35:48 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 06 Jul 2011 15:35:48 -0500
Subject: [Swift-devel] Call today
In-Reply-To: <alpine.DEB.2.00.1107061435540.1980@wozniak-desktop.mcs.anl.gov>
References: <alpine.DEB.2.00.1107061435540.1980@wozniak-desktop.mcs.anl.gov>
Message-ID: <1309984548.32472.1.camel@blabla>

Sorry, I got some more or less emergency and I can't talk today.

Mihael

On Wed, 2011-07-06 at 14:36 -0500, Justin M Wozniak wrote:
> I'll be on for a call today to touch base on the latest developments...
> 




From wilde at mcs.anl.gov  Wed Jul  6 16:30:56 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 6 Jul 2011 16:30:56 -0500 (CDT)
Subject: [Swift-devel] Call today
In-Reply-To: <CAE50RW22j+OM-SCFnXYmP8M=28oRrFQ=WnRmvUDyz2vb5KtS1A@mail.gmail.com>
Message-ID: <889831600.74813.1309987856744.JavaMail.root@zimbra.anl.gov>

We called off the call in Mihael's absence. Justin will be sending email on release plans. 

- Mike 

----- Original Message -----


i've been trying to expand the job attributes list that you started, justin: https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1 

feel free to correct/add anything or point me to code :) 


On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak < wozniak at mcs.anl.gov > wrote: 



I'll be on for a call today to touch base on the latest developments... 

-- 
Justin M Wozniak 
_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Sarah Kenny 
Programmer 
University of Chicago, Computation Institute 
University of California Irvine, Dept. of Neurology 
773-818-8300 


_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110706/f09c3e41/attachment.html>

From hategan at mcs.anl.gov  Wed Jul  6 20:39:14 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 06 Jul 2011 20:39:14 -0500
Subject: [Swift-devel] provenance
In-Reply-To: <1309886134.4432.6.camel@blabla>
References: <1309886134.4432.6.camel@blabla>
Message-ID: <1310002754.2878.2.camel@blabla>

It has been pointed out by Ketan in a bug report that there already is
an option in swift.properties related to whether provenance should be
logger or not, so I removed the flag mentioned below and changed the
code to use the property instead to control whether the kml will contain
provenance stuff.

Sorry for the mix-up.

Mihael

On Tue, 2011-07-05 at 12:15 -0500, Mihael Hategan wrote:
> In trunk provenance inside the compiled file is disabled by default. It
> is my understanding that it was more common for it to not be used than
> otherwise, but if this isn't so, let me know.
> 
> It can be enabled with "-enable.provenance" on the swift command line.
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From yadudoc1729 at gmail.com  Thu Jul  7 14:18:21 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 8 Jul 2011 00:48:21 +0530
Subject: [Swift-devel] Explicit subscript type declaration for arrays
Message-ID: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>

Hi,

This week I've been trying to implement support explicit type
declaration for array subscripts. This feature is expected to
fix the earlier issue of subscripts not matching foreach indices.
In addition we'll be able to check the type consistency of
subscripts.

Right now the following piece of code works,

int array[int][string];
array[5] << 10;
array[5] << 20;

foreach v,index in array[5]{
        trace (v,index);
        trace (array[5][index]);
}

Right now, I'm checking for the basic types already defined as
well as types found earlier on parsing(which I think is the right
behavior), but these are not tested and will take a couple of
days more.

I am pushing all the code, as soon as something works to my
git account [1]. I think this is better than pasting patches, over
mail, which will probably be out-dated fast. The latest commit
with changes enabling explicit subscript types are here[2].

Mihael had pointed out a case in which a struct was used as
subscript. How will such an array be declared ? I am blank
about how struct types. Suggestions and ideas are welcome.
( I really need help! )

[1] https://github.com/yadudoc/Swift
[2] https://github.com/yadudoc/Swift/commit/db5d2cd337e0fb06112b458fdd915659dcb2f153

-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Thu Jul  7 14:30:53 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Jul 2011 14:30:53 -0500
Subject: [Swift-devel] Explicit subscript type declaration for arrays
In-Reply-To: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>
References: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>
Message-ID: <1310067053.10567.10.camel@blabla>

On Fri, 2011-07-08 at 00:48 +0530, Yadu Nand wrote:
[...]
> 
> Right now, I'm checking for the basic types already defined as
> well as types found earlier on parsing(which I think is the right
> behavior), but these are not tested and will take a couple of
> days more.
> 
[...]
> 
> Mihael had pointed out a case in which a struct was used as
> subscript. How will such an array be declared ? I am blank
> about how struct types. Suggestions and ideas are welcome.
> ( I really need help! )

You could start with support for only primitive types as indices (i.e.
Type.isPrimitive()).

In terms of the structs, these look like.. structs:
type mystruct {
  int a;
  string b;
}

What's needed if they are to be used for array keys is an equality test,
which can be straightforwardly done in the struct case by comparing
field values recursively. Since non-primitive types cannot really be
compared properly at this time, this would probably mean that
non-primitive types should not be allowed in composite types that can be
array keys.



From yadudoc1729 at gmail.com  Thu Jul  7 15:10:52 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 8 Jul 2011 01:40:52 +0530
Subject: [Swift-devel] Explicit subscript type declaration for arrays
In-Reply-To: <1310067053.10567.10.camel@blabla>
References: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>
	<1310067053.10567.10.camel@blabla>
Message-ID: <CANa904k7qPhC1MeixXbJ=41GRtQcOJ5TzkQRZ1+MRbtCbL4z5A@mail.gmail.com>

> You could start with support for only primitive types as indices (i.e.
> Type.isPrimitive()).
Currently, this is what I'm doing. Though the logic I'm using requires
listing out the types listed under the hashmap types in  types.java.
I'll post behavior on cases where types defined earlier (besides the
primitives) are used as subscripts.

> In terms of the structs, these look like.. structs:
> type mystruct {
> ?int a;
> ?string b;
> }
>
> What's needed if they are to be used for array keys is an equality test,
> which can be straightforwardly done in the struct case by comparing
> field values recursively. Since non-primitive types cannot really be
> compared properly at this time, this would probably mean that
> non-primitive types should not be allowed in composite types that can be
> array keys.

Will it be any different if whenever we use a composite type we create
a string hashcode as the key ? Will this not eliminate the issue of
comparison ? I'm not sure if we can create a hashcode, and I bet the
subscript type issues are going to be weird with the declaration pointing
to a type and the actual type being a string.

Is the following the expected syntax in this case ?

type my_struct {
    int num ;
    string name;
}

int array [ int ] [ my_struct ];

-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Thu Jul  7 15:42:45 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 07 Jul 2011 15:42:45 -0500
Subject: [Swift-devel] Explicit subscript type declaration for arrays
In-Reply-To: <CANa904k7qPhC1MeixXbJ=41GRtQcOJ5TzkQRZ1+MRbtCbL4z5A@mail.gmail.com>
References: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>
	<1310067053.10567.10.camel@blabla>
	<CANa904k7qPhC1MeixXbJ=41GRtQcOJ5TzkQRZ1+MRbtCbL4z5A@mail.gmail.com>
Message-ID: <1310071365.10567.15.camel@blabla>

On Fri, 2011-07-08 at 01:40 +0530, Yadu Nand wrote:
> > You could start with support for only primitive types as indices (i.e.
> > Type.isPrimitive()).
> Currently, this is what I'm doing. Though the logic I'm using requires
> listing out the types listed under the hashmap types in  types.java.
> I'll post behavior on cases where types defined earlier (besides the
> primitives) are used as subscripts.
> 
> > In terms of the structs, these look like.. structs:
> > type mystruct {
> >  int a;
> >  string b;
> > }
> >
> > What's needed if they are to be used for array keys is an equality test,
> > which can be straightforwardly done in the struct case by comparing
> > field values recursively. Since non-primitive types cannot really be
> > compared properly at this time, this would probably mean that
> > non-primitive types should not be allowed in composite types that can be
> > array keys.
> 
> Will it be any different if whenever we use a composite type we create
> a string hashcode as the key ?

If you can guarantee that for every two different structs the hash codes
are different then it's only a matter of which implementation is
simpler/more clear.

>  Will this not eliminate the issue of
> comparison ?

No. It is a solution to the issue of comparison. Though I suspect it's
not a very good one.

>  I'm not sure if we can create a hashcode, and I bet the
> subscript type issues are going to be weird with the declaration pointing
> to a type and the actual type being a string.
> 
> Is the following the expected syntax in this case ?
> 
> type my_struct {
>     int num ;
>     string name;
> }
> 
> int array [ int ] [ my_struct ];
> 

Yes.



From dsk at ci.uchicago.edu  Fri Jul  8 09:48:24 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Fri, 8 Jul 2011 09:48:24 -0500
Subject: [Swift-devel] Fwd: [Colloquium] Shaw/Dissertation Defense/Jul 22,
	2011
References: <20110708144536.62293EBD9@traell.cs.uchicago.edu>
Message-ID: <2D49B534-1455-4696-9B77-502BD2A1C2CE@ci.uchicago.edu>

This might be interesting to Swift people...

Begin forwarded message:

> From: "Margaret Jaffey" <margaret at cs.uchicago.edu>
> Date: July 8, 2011 9:45:36 AM CDT
> To: cs at mailman.cs.uchicago.edu, colloquium at mailman.cs.uchicago.edu
> Subject: [Colloquium] Shaw/Dissertation Defense/Jul 22, 2011
> 
> 
> 
>       Department of Computer Science/The University of Chicago
> 
>                     *** Dissertation Defense ***
> 
> 
> Candidate:  Adam Shaw
> 
> Date:  Friday, July 22, 2011
> 
> Time:  10:00 AM
> 
> Place:  Ryerson 277
> 
> Title: Implementation Techniques for Nested Data-Parallel Languages
> 
> Abstract:
> Nested data-parallel languages allow computation in parallel over
> irregular nested data structures. The classic approach to compiling
> nested data parallelism in high-level languages is to apply flattening
> to nested structures. Flattening separates nested data and its shape
> into distinct values: a flat data vector, and a representation of the
> nesting information. In a parallel context, flattening is beneficial
> because computation on flat data vectors maps easily onto parallel
> hardware, and it is easier to partition work across processing
> elements in flattened code.
> 
> Traditionally, flattening is a wholesale transformation that unravels
> all nested data structures and correspondingly transforms the
> operations on them. Such total flattening may not always yield best
> performance: sometimes we might want to flatten part way, or not at
> all. To accommodate such possibilities, we present hybrid flattening.
> In hybrid flattening transformations, only certain structures are
> flattened, and to varying degrees. This dissertation presents a formal
> framework for defining hybrid flattening transformations.
> 
> We use our framework to define a novel flattening transformation on a
> model programming language. Guided by our model, we implemented our
> transformation in the compiler for Parallel ML, a nested data-parallel
> language with implicitly-threaded features. Our implementation
> demonstrates the utility of the transformation. Across various
> benchmarks, transformed programs perform better than untransformed
> ones, scale better, and compete favorably against efficient sequential
> programs in C and SML. With our system, running PML programs on a
> 48-core machine yields as much as a thirtyfold improvement over their
> sequential counterparts.
> 
> Adam's advisor is Prof. John Reppy
> 
> Login to the Computer Science Department website for details,
> including a draft copy of the dissertation:
> 
> https://www.cs.uchicago.edu/phd/phd_announcements#adamshaw
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Margaret P. Jaffey            margaret at cs.uchicago.edu
> Department of Computer Science
> Student Support Rep (Ry 156)               (773) 702-6011
> The University of Chicago      http://www.cs.uchicago.edu
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> _______________________________________________
> Colloquium mailing list  -  Colloquium at mailman.cs.uchicago.edu
> https://mailman.cs.uchicago.edu/mailman/listinfo/colloquium

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110708/00620fa2/attachment.html>

From yadudoc1729 at gmail.com  Fri Jul  8 14:00:39 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Sat, 9 Jul 2011 00:30:39 +0530
Subject: [Swift-devel] Explicit subscript type declaration for arrays
In-Reply-To: <1310071365.10567.15.camel@blabla>
References: <CANa904nP_meFKev4hp4wY3-9EH2gt352E_PnYdBGf2i3thNMPA@mail.gmail.com>
	<1310067053.10567.10.camel@blabla>
	<CANa904k7qPhC1MeixXbJ=41GRtQcOJ5TzkQRZ1+MRbtCbL4z5A@mail.gmail.com>
	<1310071365.10567.15.camel@blabla>
Message-ID: <CANa904=BEZeYFhMj1QM+K+2xcmRJW-yEPz=g2+9f7VaBLikG2g@mail.gmail.com>

Hi,

There was a bug in the code which did not allow arrays of more than two
dimensions. This has been fixed. Explicit subscript types are being
pushed to a branch on my git account [1].

Fair warning, that its very buggy right now, and a lot of checks placed
earlier are disabled.

I'm trying to add support for types(structs) as subscripts.

[1] https://github.com/yadudoc/Swift/tree/subs_type_experimental


-- 
Thanks and Regards,
Yadu Nand B


From iraicu at cs.iit.edu  Mon Jul 11 09:24:28 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Mon, 11 Jul 2011 09:24:28 -0500
Subject: [Swift-devel] CFP: The 12th IEEE/ACM International Conference on
 Grid Computing (Grid2011)
Message-ID: <4E1B079C.3040801@cs.iit.edu>

Call for Papers

The 12th IEEE/ACM International Conference on Grid Computing (Grid2011)
Lyon, France
Sep 21 - Sep 23, 2011
http://grid2011.mnm-team.org/
Co-located with the EGI Technical Forum and OGF

Sponsored by:
  - The IEEE Computer Society Technical Committee on Scalable Computing (pending)
  - Association for Computing Machinery Special Interest Group on Computer Architecture (pending)

Grid computing enables the sharing of distributed computing and data
resources such as processing, network bandwidth and storage capacity
to create a cohesive resource environment for executing distributed
applications. The Grid conference series is an annual international
meeting that brings together a community of researchers, developers,
practitioners, and users involved with Grid technology. The objective
of the meeting is to serve as both the premier venue for presenting
foremost research results in the area and as a forum for introducing
and exploring new concepts. The conference will feature invited talks,
workshops, and refereed paper presentations.

Grid 2011 welcomes paper and poster submissions on innovative work
from researchers in academia, industry and government describing
original research work in grid computing. Previous events in this
series have been successful in attracting high quality papers and a
wide international participation. This event will be co-located with
the EGI Technical Forum.

SCOPE

Grid 2011 topics of interest include, but are not limited to:

- Applications, including eScience and eBusiness Applications
- Architectures and Fabrics
- Authentication, Authorization, Auditing and Accounting
- Autonomic and Utility Computing on Global Grids
- Cloud computing
- Cloud, Cluster and Grid Integration
- Creation and Management of Virtual Enterprises and Organizations
- Critical surveys or reflections on the past decade on grid and distributed computing
- Dynamic, Distributed, Data-Intensive Access, Management and Processing
- Energy Efficiency and Grid
- Grid Economy and Business Models
- Infrastructure and Practise of Distributed Computing
- Metadata, Ontologies, and Provenance
- Middleware and Toolkits
- Monitoring, Management and Organization Tools
- Networking
- Performance Measurement and Modelling
- Problem Solving Environments
- Programming Models, Tools and Environments
- Production Cyberinfrastructure
- QoS and SLA Negotiation
- Resource Management, Scheduling, and Runtime Environments
- Scientific, Industrial and Social Implications
- Semantic Grid
- Standardization efforts in Grid
- Virtualization and grid computing

TECHNICAL PAPERS

Grid 2011 invites authors to submit original papers (not published or
under review elsewhere). Papers should be no more than 8 pages in
length (including diagrams and references) and be submitted as a PDF
file by using the submission system: URL

Submission implies the willingness of at least one of the authors to
register and present the paper. A separate conference proceedings will
be published and will also be a part of the IEEE Xplore and the CS
digital library. For author instructions visit www.grid2011.org.

IMPORTANT DATES

15 July   2011 Technical Paper Submission Due
17 August 2011 Paper Acceptance Notifications
24 August 2011 Full and Revised papers due
15 August 2011 Poster submissions Due
25 August 2011 Poster Acceptance Notifications

CONFERENCE ORGANISATION

General Chair: Nils gentschen Felde, MNM, Munich
Program Chair: Shantenu Jha, Rutgers, USA and Edinburgh, UK
Local Chair: Frederic Suter, IN2P3, Lyon
Workshop Chair&  Poster Chair: Gilles Fedak, INRIA, Lyon
Proceedings and Publications Chair: Rajkumar Buyya, The University of
Melbourne and Manjrasoft, Australia

Program Vice Chairs:

Clouds and Virtualisation: Roger Barga, Microsoft Research, US
Distributed Production Cyberinfrastructure and Middleware: Andrew Grimshaw, Univ. of Virginia, US
e-Research and Applications: Daniel S. Katz, Univ. of Chicago and Argonne National Lab, US
Tools&  Services, Resource Management&  Runtime Environments: Ramin Yahyapour, Dortmund
Distributed Data-Intensive Science and Systems:  Erwin Laure, KTH, Sweeden

Publicity Chairs:
Suraj Pandey, Univ of Melbourne, Australia
Yoshiyuki Watase,  KEK, Japan
Cameron Kiddle, Calgary, Canada
Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA
Adam Barker, St. Andrew's, UK

CONFERENCE ORGANISATION

General Chair: Nils gentschen Felde, MNM, Munich
Program Chair: Shantenu Jha, Rutgers, USA and Edinburgh, UK
Local Chair: Frederic Suter, IN2P3, Lyon
Workshop Chair&  Poster Chair: Gilles Fedak, INRIA, Lyon
Proceedings and Publications Chair: Rajkumar Buyya, The University of
Melbourne and Manjrasoft, Australia

Program Vice Chairs:

Clouds and Virtualisation: Roger Barga, Microsoft Research, US
Distributed Production Cyberinfrastructure and Middleware: Andrew Grimshaw, Univ. of Virginia, US
e-Research and Applications: Daniel S. Katz, Univ. of Chicago and Argonne National Lab, US
Tools&  Services, Resource Management&  Runtime Environments: Ramin Yahyapour, Dortmund
Distributed Data-Intensive Science and Systems:  Erwin Laure, KTH, Sweeden

Publicity Chairs:
Suraj Pandey, Univ of Melbourne, Australia
Yoshiyuki Watase,  KEK, Japan
Cameron Kiddle, Calgary, Canada
Ioan Raicu, Illinois Institute of Technology and Argonne National Laboratory, USA
Adam Barker, St. Andrew's, UK

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================




From ketancmaheshwari at gmail.com  Mon Jul 11 12:16:45 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Mon, 11 Jul 2011 12:16:45 -0500
Subject: [Swift-devel] Separate swift.config properties for different
	sites.xml entries
Message-ID: <CAMUuvirtrxtZragw=GUVrhc0BFi7pQ1OAR-1+-TCq9uMy8WjbQ@mail.gmail.com>

Hello,

While trying to run a cybershake workflow on ranger and localhost, I am
trying to use 2 different providers and the staging method for them are
different:

For the local provider the provider.staging should be false while for the
persistent coasters provider it should be true.

Is there a way so that can I associate the swift.properties separately for
different providers in sites.xml?

Thanks,

-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110711/eeffd629/attachment.html>

From jonmon at utexas.edu  Mon Jul 11 14:30:34 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Mon, 11 Jul 2011 14:30:34 -0500
Subject: [Swift-devel] NumberFormatException
Message-ID: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>

Mihael,
   I am getting this error using release 0.92.1.

2011-07-11 19:24:44,395+0000 INFO  unknown RUNID id=run:20110711-1924-c944yl9c
2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @ script.kml, line: 69: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
Caused by: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
	at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
	at org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
	at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
	at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
	at org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
	at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
	at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
	at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
	at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
	at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
	at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
	at org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
	at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
	at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
	at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
	at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
	at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:493)
	at java.lang.Integer.parseInt(Integer.java:514)
	at org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
	at org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
	at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
	at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
	... 63 more

The files needed for this run are located in ~jonmon/run.0000 on the ci machines.

From hategan at mcs.anl.gov  Mon Jul 11 17:22:56 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 11 Jul 2011 17:22:56 -0500
Subject: [Swift-devel] passive workers scheduler throttle
Message-ID: <1310422976.16228.5.camel@blabla>

trunk cog r3179/swift r4816 supports the ability to scale the throttle
for a site based on the number of workers running on a site.

To enable it you need to have the passive worker manager and the
globus:throttleTracksWorkers profile set to "true". E.g.

    <profile namespace="globus" key="workerManager">passive</profile>
    <profile namespace="globus"
key="throttleTracksWorkers">true</profile>

One thing to mention is that while the throttle will match the number of
workers, it will never go below 1. That one job is needed to
start/connect to the service in the current scheme of things.

One other thing to mention was that there was a bit of trouble with
passive workers losing a connection to the service. It would typically
result in the jobs that were running on the lost workers to appear to
never complete. That has now been fixed.

Mihael



From jonmon at utexas.edu  Mon Jul 11 22:45:46 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Mon, 11 Jul 2011 22:45:46 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
Message-ID: <1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>

I actually meant to send this to swift-devel.

Here is the background to the problem.  I have data on PADS, I am executing Swift on a VM, and I want to use OSG to compute with the data.  Before Mike left for vacation he said that you can map data in Swift using the GSIURI scheme but he did not tell me how.  What I did below is 

file data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;

This does not seem to work in release 0.92.1 as the error below shows.  How do you map data in Swift using the GSIURI scheme?

Begin forwarded message:

> From: Jonathan Monette <jonmon at utexas.edu>
> Date: July 11, 2011 2:30:34 PM CDT
> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> Subject: NumberFormatException
> 
> Mihael,
>   I am getting this error using release 0.92.1.
> 
> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID id=run:20110711-1924-c944yl9c
> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @ script.kml, line: 69: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
> java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
> Caused by: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
> 	at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> 	at org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> 	at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> 	at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> 	at org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> 	at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> 	at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> 	at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> 	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> 	at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> 	at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 	at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> 	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 	at org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 	at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> 	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 	at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> 	at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> 	at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> 	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> 	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> 	at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> 	at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
> 	at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
> 	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
> 	at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.NumberFormatException: For input string: ""
> 	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> 	at java.lang.Integer.parseInt(Integer.java:493)
> 	at java.lang.Integer.parseInt(Integer.java:514)
> 	at org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> 	at org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> 	at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> 	at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> 	... 63 more
> 
> The files needed for this run are located in ~jonmon/run.0000 on the ci machines.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110711/57f0d8c0/attachment.html>

From hategan at mcs.anl.gov  Tue Jul 12 02:16:16 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Jul 2011 02:16:16 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
	<1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
Message-ID: <1310454976.19717.2.camel@blabla>

You have a colon after the host name but no port. Either remove the
colon or put a number after it.

On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
> I actually meant to send this to swift-devel.
> 
> 
> Here is the background to the problem.  I have data on PADS, I am
> executing Swift on a VM, and I want to use OSG to compute with the
> data.  Before Mike left for vacation he said that you can map data in
> Swift using the GSIURI scheme but he did not tell me how.  What I did
> below is 
> 
> 
> file
> data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> 
> 
> This does not seem to work in release 0.92.1 as the error below
> shows.  How do you map data in Swift using the GSIURI scheme?
> 
> Begin forwarded message:
> 
> > From: Jonathan Monette <jonmon at utexas.edu>
> > 
> > Date: July 11, 2011 2:30:34 PM CDT
> > 
> > To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> > 
> > Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> > 
> > Subject: NumberFormatException
> > 
> > 
> > Mihael,
> >   I am getting this error using release 0.92.1.
> > 
> > 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
> > id=run:20110711-1924-c944yl9c
> > 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
> > script.kml, line: 69: java.lang.RuntimeException:
> > java.lang.NumberFormatException: For input string: ""
> > java.lang.RuntimeException: java.lang.NumberFormatException: For
> > input string: ""
> > Caused by: java.lang.RuntimeException:
> > java.lang.NumberFormatException: For input string: ""
> > at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> > at
> > org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> > at
> > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> > at
> > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> > at
> > org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> > at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> > at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> > at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> > at
> > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at
> > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > at
> > org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> > at
> > org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> > at
> > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> > at
> > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > at edu.emory.mathcs.backport.java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:431)
> > at
> > edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:643)
> > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:668)
> > at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.lang.NumberFormatException: For input string: ""
> > at
> > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > at java.lang.Integer.parseInt(Integer.java:493)
> > at java.lang.Integer.parseInt(Integer.java:514)
> > at
> > org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> > at
> > org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> > at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> > at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> > ... 63 more
> > 
> > The files needed for this run are located in ~jonmon/run.0000 on the
> > ci machines.
> 
> 




From hategan at mcs.anl.gov  Tue Jul 12 02:20:53 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Jul 2011 02:20:53 -0500
Subject: [Swift-devel] Separate swift.config properties for different
 sites.xml entries
In-Reply-To: <CAMUuvirtrxtZragw=GUVrhc0BFi7pQ1OAR-1+-TCq9uMy8WjbQ@mail.gmail.com>
References: <CAMUuvirtrxtZragw=GUVrhc0BFi7pQ1OAR-1+-TCq9uMy8WjbQ@mail.gmail.com>
Message-ID: <1310455253.19804.1.camel@blabla>

On Mon, 2011-07-11 at 12:16 -0500, Ketan Maheshwari wrote:
> Hello,
> 
> While trying to run a cybershake workflow on ranger and localhost, I
> am trying to use 2 different providers and the staging method for them
> are different:
> 
> For the local provider the provider.staging should be false while for
> the persistent coasters provider it should be true.
> 
> Is there a way so that can I associate the swift.properties separately
> for different providers in sites.xml?

No, but the local provider supports provider staging.




From ketancmaheshwari at gmail.com  Tue Jul 12 14:16:52 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 12 Jul 2011 14:16:52 -0500
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
Message-ID: <CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>

Mihael,

I tried to further investigate the issue and from the logs it seems that
Swift is trying to execute the mkoffset app before creating a jobs/
directory in workdir. Could it be that this is an ordering issue. For
instance, I see the following line:

2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2 JOB_START
jobid=mkoffset-mex8fvck tr=mkoffset arguments=[200.0, 60.0]
tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
host=localhost

but do not see a createdir corresponding to above.

I have ran this workflow successfully with 0.92.1 so, I am pretty sure that
it works correctly as far as order of execution is concerned.

Thanks for any more insights into this.

Regards,
Ketan

---------- Forwarded message ----------
From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
Date: Mon, Jul 11, 2011 at 10:31 PM
Subject: trunk FileNotFoundException
To: swift-user at ci.uchicago.edu


Hello,

Using Swift trunk, I am running the SCEC workflow from Communicado using
ranger, localhost and OSG resources.

One particular app 'mkoffset' which is destined to run on localhost is
faulting with FileNotFoundException.

The log does give information on its mapping and when it gets 'cleared'.

The config, tc, sites and log files for the run could be found here:
http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log is 90M, upload size
exceeded!)

The error stack that I am getting on stdout is:

Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500  Selecting site:390  Stage
in:16  Active:9  Checking status:1  Finished successfully:36 Failed but can
retry:3
org.globus.cog.abstraction.impl.file.FileNotFoundException: File not found:
/var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
    at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
    at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
    at
org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
    at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
    at
org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
    at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
    at java.lang.Thread.run(Thread.java:619)
org.globus.cog.abstraction.impl.file.FileNotFoundException: File not found:
/var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
    at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
    at
org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
    at
org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
    at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
    at
org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
    at
org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
    at java.lang.Thread.run(Thread.java:619)
Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500  Selecting site:389  Stage
in:16  Active:9  Checking status:1  Finished successfully:38 Failed but can
retry:4
Execution failed:
    java.lang.NullPointerException
    at
org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
    at org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
    at
org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
    at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at
org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)


Any clues?

Thanks,
-- 
Ketan





-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/f7005289/attachment.html>

From dsk at ci.uchicago.edu  Tue Jul 12 15:00:32 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Tue, 12 Jul 2011 15:00:32 -0500
Subject: [Swift-devel] somewhat interesting paper
References: <AEE34A0C-DCF4-4D85-8E93-2FD1171068E9@uchicago.edu>
Message-ID: <C0D9F78A-A217-4424-A23F-761C92785592@ci.uchicago.edu>

From Aaron Dinner...

Begin forwarded message:

> From: Aaron Dinner <dinner at uchicago.edu>
> Date: July 12, 2011 2:51:01 PM CDT
> To: "Daniel S. Katz" <dsk at ci.uchicago.edu>
> 

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/439be084/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Malawski2010.pdf
Type: application/pdf
Size: 5386547 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/439be084/attachment.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/439be084/attachment-0001.html>

From iraicu at cs.iit.edu  Tue Jul 12 15:52:49 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Tue, 12 Jul 2011 15:52:49 -0500
Subject: [Swift-devel] CFP: The 21st Int. ACM Symp. on High-Performance
 Parallel and Distributed Computing (HPDC'12) 2012
Message-ID: <4E1CB421.4070701@cs.iit.edu>

                       **** CALL FOR PAPERS ****

                  **** CALL FOR WORKSHOP PROPOSALS ****

                 The 21st International ACM Symposium on
          High-Performance Parallel and Distributed Computing
                             (HPDC'12)

           Delft University of Technology, Delft, the Netherlands

                          June 18-22, 2012

                     http://www.hpdc.org/2012


The ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC)
is the premier annual conference on the design, the implementation, the evaluation, and
the use of parallel and distributed systems for high-end computing. HPDC'12 will take place
in Delft, the Netherlands, a historical, picturesque city that is less than one hour away
from Amsterdam-Schiphol airport. The conference will be held on June 20-22 (Wednesday to
Friday), with affiliated workshops taking place on June 18-19 (Monday and Tuesday).


**** SUBMISSION DEADLINES ****
Abstracts:    16 January 2012
Papers:       23 January 2012 (No extensions!)


**** HPDC'12 GENERAL CHAIR ****
Dick Epema, Delft University of Technology, Delft, the Netherlands


**** HPDC'12 PROGRAM CO-CHAIRS ****
Thilo Kielmann, Vrije Universiteit, Amsterdam, the Netherlands
Matei Ripeanu, The University of British Columbia, Vancouver, Canada


**** HPDC'12 WORKSHOPS CHAIR ****
Alexandru Iosup, Delft University of Technology, Delft, the Netherlands


**** SCOPE AND TOPICS ****
Submissions are welcomed on all forms of high-performance parallel and distributed computing,
including but not limited to clusters, clouds, grids, utility computing, data-intensive
computing, and massively multicore systems. Submissions that explore solutions to estimate
and reduce the energy footprint of such systems are particularly encouraged. All papers
will be evaluated for their originality, potential impact, correctness, quality of
presentation, appropriate presentation of related work, and relevance to the conference,
with a strong preference for rigorous results obtained in operational parallel and
distributed systems.

The topics of interest of the conference include, but are not limited to, the following,
in the context of high-performance parallel and distributed computing:

    - Systems, networks, and architectures for high-end computing
    - Massively multicore systems
    - Virtualization of machines, networks, and storage
    - Programming languages and environments
    - I/O, storage systems, and data management
    - Resource management, energy and cost minimizations
    - Performance modeling and analysis
    - Fault tolerance, reliability, and availability
    - Data-intensive computing
    - Applications of parallel and distributed computing


**** PAPER SUBMISSION GUIDELINES ****
Authors are invited to submit technical papers of at most 12 pages in PDF format, including
figures and references. Papers should be formatted in the ACM Proceedings Style and
submitted via the conference web site. No changes to the margins, spacing, or font sizes as
specified by the style file are allowed. Accepted papers will appear in the conference
proceedings, and will be incorporated into the ACM Digital Library. A limited number of
papers will be accepted as posters.

Papers must be self-contained and provide the technical substance required for the program
committee to evaluate their contributions. Submitted papers must be original work that has
not appeared in and is not under consideration for another conference or a journal. See the
ACM Prior Publication Policy for more details.


**** IMPORTANT DATES ****
Workshop Proposals Due:			3 October 2011
Abstracts Due:				16 January 2012
Papers Due:				23 January 2012 (No extensions!)
Reviews Released to Authors:		8 March 2012
Author Rebuttals Due:			12 March 2012
Author Notifications:			19 March 2012
Final Papers Due:			16 April 2012
Conference Dates:			18-22 June 2012


**** CALL FOR WORKSHOP PROPOSALS ****
Workshops affiliated with HPDC will be held on June 18-19 (Monday and Tuesday). For more
information on the workshops and for the complete Call for Workshop Proposals, see the
workshops page on the conference website.

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================




From jonmon at utexas.edu  Tue Jul 12 15:59:43 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Tue, 12 Jul 2011 15:59:43 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <1310454976.19717.2.camel@blabla>
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
	<1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
	<1310454976.19717.2.camel@blabla>
Message-ID: <D6303CB9-22C9-4ACC-AF81-C6B67807D3C1@utexas.edu>

How does this syntax look when mapping with the GSIURI?  How does it work?  I am still getting errors when trying to map a file with a GSIURI.  It says file not found.  Maybe I have a different understanding on how it works.

The files and the logs are in ~jonmon/run.0001 on the ci machines.

On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:

> You have a colon after the host name but no port. Either remove the
> colon or put a number after it.
> 
> On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
>> I actually meant to send this to swift-devel.
>> 
>> 
>> Here is the background to the problem.  I have data on PADS, I am
>> executing Swift on a VM, and I want to use OSG to compute with the
>> data.  Before Mike left for vacation he said that you can map data in
>> Swift using the GSIURI scheme but he did not tell me how.  What I did
>> below is 
>> 
>> 
>> file
>> data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
>> 
>> 
>> This does not seem to work in release 0.92.1 as the error below
>> shows.  How do you map data in Swift using the GSIURI scheme?
>> 
>> Begin forwarded message:
>> 
>>> From: Jonathan Monette <jonmon at utexas.edu>
>>> 
>>> Date: July 11, 2011 2:30:34 PM CDT
>>> 
>>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
>>> 
>>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
>>> 
>>> Subject: NumberFormatException
>>> 
>>> 
>>> Mihael,
>>>  I am getting this error using release 0.92.1.
>>> 
>>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
>>> id=run:20110711-1924-c944yl9c
>>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
>>> script.kml, line: 69: java.lang.RuntimeException:
>>> java.lang.NumberFormatException: For input string: ""
>>> java.lang.RuntimeException: java.lang.NumberFormatException: For
>>> input string: ""
>>> Caused by: java.lang.RuntimeException:
>>> java.lang.NumberFormatException: For input string: ""
>>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
>>> at
>>> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
>>> at
>>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
>>> at
>>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
>>> at
>>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
>>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
>>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
>>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
>>> at
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>> at
>>> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
>>> at edu.emory.mathcs.backport.java.util.concurrent.Executors
>>> $RunnableAdapter.call(Executors.java:431)
>>> at
>>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>>> $Worker.runTask(ThreadPoolExecutor.java:643)
>>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>>> $Worker.run(ThreadPoolExecutor.java:668)
>>> at java.lang.Thread.run(Thread.java:636)
>>> Caused by: java.lang.NumberFormatException: For input string: ""
>>> at
>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>> at java.lang.Integer.parseInt(Integer.java:493)
>>> at java.lang.Integer.parseInt(Integer.java:514)
>>> at
>>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
>>> at
>>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
>>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
>>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
>>> ... 63 more
>>> 
>>> The files needed for this run are located in ~jonmon/run.0000 on the
>>> ci machines.
>> 
>> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From skenny at uchicago.edu  Tue Jul 12 16:59:33 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Tue, 12 Jul 2011 14:59:33 -0700
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <D6303CB9-22C9-4ACC-AF81-C6B67807D3C1@utexas.edu>
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
	<1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
	<1310454976.19717.2.camel@blabla>
	<D6303CB9-22C9-4ACC-AF81-C6B67807D3C1@utexas.edu>
Message-ID: <CAE50RW1-MmrDpbqB7kBWhXJ04Jy4U2UJiLgWs=K=Ummc6j8yYw@mail.gmail.com>

i've used this successfully in the past:

file mybrain<single_file_mapper; file="gsiftp://
calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;

it's been a while though...haven't tested with the latest swift...

On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette <jonmon at utexas.edu> wrote:

> How does this syntax look when mapping with the GSIURI?  How does it work?
>  I am still getting errors when trying to map a file with a GSIURI.  It says
> file not found.  Maybe I have a different understanding on how it works.
>
> The files and the logs are in ~jonmon/run.0001 on the ci machines.
>
> On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:
>
> > You have a colon after the host name but no port. Either remove the
> > colon or put a number after it.
> >
> > On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
> >> I actually meant to send this to swift-devel.
> >>
> >>
> >> Here is the background to the problem.  I have data on PADS, I am
> >> executing Swift on a VM, and I want to use OSG to compute with the
> >> data.  Before Mike left for vacation he said that you can map data in
> >> Swift using the GSIURI scheme but he did not tell me how.  What I did
> >> below is
> >>
> >>
> >> file
> >> data<"gsiftp://stor01.pads.ci.uchicago.edu:
> /gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> >>
> >>
> >> This does not seem to work in release 0.92.1 as the error below
> >> shows.  How do you map data in Swift using the GSIURI scheme?
> >>
> >> Begin forwarded message:
> >>
> >>> From: Jonathan Monette <jonmon at utexas.edu>
> >>>
> >>> Date: July 11, 2011 2:30:34 PM CDT
> >>>
> >>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> >>>
> >>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> >>>
> >>> Subject: NumberFormatException
> >>>
> >>>
> >>> Mihael,
> >>>  I am getting this error using release 0.92.1.
> >>>
> >>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
> >>> id=run:20110711-1924-c944yl9c
> >>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
> >>> script.kml, line: 69: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> java.lang.RuntimeException: java.lang.NumberFormatException: For
> >>> input string: ""
> >>> Caused by: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> >>> at
> >>>
> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> >>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> >>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> >>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>>
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.Executors
> >>> $RunnableAdapter.call(Executors.java:431)
> >>> at
> >>>
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.runTask(ThreadPoolExecutor.java:643)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.run(ThreadPoolExecutor.java:668)
> >>> at java.lang.Thread.run(Thread.java:636)
> >>> Caused by: java.lang.NumberFormatException: For input string: ""
> >>> at
> >>>
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >>> at java.lang.Integer.parseInt(Integer.java:493)
> >>> at java.lang.Integer.parseInt(Integer.java:514)
> >>> at
> >>>
> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> >>> at
> >>>
> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> >>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> >>> ... 63 more
> >>>
> >>> The files needed for this run are located in ~jonmon/run.0000 on the
> >>> ci machines.
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/2b137bc8/attachment.html>

From jonmon at utexas.edu  Tue Jul 12 17:14:32 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Tue, 12 Jul 2011 17:14:32 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <CAE50RW1-MmrDpbqB7kBWhXJ04Jy4U2UJiLgWs=K=Ummc6j8yYw@mail.gmail.com>
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
	<1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
	<1310454976.19717.2.camel@blabla>
	<D6303CB9-22C9-4ACC-AF81-C6B67807D3C1@utexas.edu>
	<CAE50RW1-MmrDpbqB7kBWhXJ04Jy4U2UJiLgWs=K=Ummc6j8yYw@mail.gmail.com>
Message-ID: <79194715-73C7-4A7D-BAAE-DFBDA17FCC87@utexas.edu>

Thanks.  I just verified that something maybe wrong with the path.  I could not globus-url-copy that URI so I will continue to investigate

On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:

> i've used this successfully in the past:
> 
> file mybrain<single_file_mapper; file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> 
> it's been a while though...haven't tested with the latest swift...
> 
> On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette <jonmon at utexas.edu> wrote:
> How does this syntax look when mapping with the GSIURI?  How does it work?  I am still getting errors when trying to map a file with a GSIURI.  It says file not found.  Maybe I have a different understanding on how it works.
> 
> The files and the logs are in ~jonmon/run.0001 on the ci machines.
> 
> On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:
> 
> > You have a colon after the host name but no port. Either remove the
> > colon or put a number after it.
> >
> > On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
> >> I actually meant to send this to swift-devel.
> >>
> >>
> >> Here is the background to the problem.  I have data on PADS, I am
> >> executing Swift on a VM, and I want to use OSG to compute with the
> >> data.  Before Mike left for vacation he said that you can map data in
> >> Swift using the GSIURI scheme but he did not tell me how.  What I did
> >> below is
> >>
> >>
> >> file
> >> data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> >>
> >>
> >> This does not seem to work in release 0.92.1 as the error below
> >> shows.  How do you map data in Swift using the GSIURI scheme?
> >>
> >> Begin forwarded message:
> >>
> >>> From: Jonathan Monette <jonmon at utexas.edu>
> >>>
> >>> Date: July 11, 2011 2:30:34 PM CDT
> >>>
> >>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> >>>
> >>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> >>>
> >>> Subject: NumberFormatException
> >>>
> >>>
> >>> Mihael,
> >>>  I am getting this error using release 0.92.1.
> >>>
> >>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
> >>> id=run:20110711-1924-c944yl9c
> >>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
> >>> script.kml, line: 69: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> java.lang.RuntimeException: java.lang.NumberFormatException: For
> >>> input string: ""
> >>> Caused by: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> >>> at
> >>> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> >>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> >>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> >>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.Executors
> >>> $RunnableAdapter.call(Executors.java:431)
> >>> at
> >>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.runTask(ThreadPoolExecutor.java:643)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.run(ThreadPoolExecutor.java:668)
> >>> at java.lang.Thread.run(Thread.java:636)
> >>> Caused by: java.lang.NumberFormatException: For input string: ""
> >>> at
> >>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >>> at java.lang.Integer.parseInt(Integer.java:493)
> >>> at java.lang.Integer.parseInt(Integer.java:514)
> >>> at
> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> >>> at
> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> >>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> >>> ... 63 more
> >>>
> >>> The files needed for this run are located in ~jonmon/run.0000 on the
> >>> ci machines.
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> -- 
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/8ded57c2/attachment.html>

From ketancmaheshwari at gmail.com  Tue Jul 12 17:47:56 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 12 Jul 2011 17:47:56 -0500
Subject: [Swift-devel] trunk FileNotFoundException
In-Reply-To: <CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
Message-ID: <CAMUuvir=jzcfbH3WH4oP8YRdn6z0=0CxOhEX0kgfqf+TSgnKjw@mail.gmail.com>

Some updates on this,

I tried the same workflow with same setting on 0.92.1 and I was getting the
"variable already in cache" message that I found from the archive is because
of a remapping of an actively mapped variable. That error went away when I
used concurrent_mapper instead of single_file_mapper.

-  //offset_file file <single_file_mapper;
file=@strcat("LGU/offset-",_size)>;
+ offset_file file <concurrent_mapper; location="LGU", prefix="offset-",
suffix=_size>;

With the above mod, I ran the same workflow with Swift trunk and seems the
java FileNotFoundException is gone. This seems to be a manifestation of the
same "var already in cache" bug of 0.92.

While the above issue does not appear anymore, the workflow still abruptly
gets halted with java.lang.NullPointerException. A complete stack is as
follows:

Progress:  time: Tue, 12 Jul 2011 17:30:10 -0500  Selecting site:402  Stage
in:12  Active:5  Stage out:1  Finished successfully:19
Execution failed:
    java.lang.NullPointerException
    at
org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
    at org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
    at
org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
    at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at
org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at
org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
    at
org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
    at
org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
    at
org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29)
    at
org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20)
    at
org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139)
    at
org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197)
    at
org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104)
    at
org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)


I see the same exception on the log. I could not find any indications as to
what is causing this.

The log for this run can be found here:
http://www.ci.uchicago.edu/~ketan/files/postproc-20110712-1723-3uhgk3i6.log

--
Ketan

On Tue, Jul 12, 2011 at 2:16 PM, Ketan Maheshwari <
ketancmaheshwari at gmail.com> wrote:

> Mihael,
>
> I tried to further investigate the issue and from the logs it seems that
> Swift is trying to execute the mkoffset app before creating a jobs/
> directory in workdir. Could it be that this is an ordering issue. For
> instance, I see the following line:
>
> 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2 JOB_START
> jobid=mkoffset-mex8fvck tr=mkoffset arguments=[200.0, 60.0]
> tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
> host=localhost
>
> but do not see a createdir corresponding to above.
>
> I have ran this workflow successfully with 0.92.1 so, I am pretty sure that
> it works correctly as far as order of execution is concerned.
>
> Thanks for any more insights into this.
>
> Regards,
> Ketan
>
>
> ---------- Forwarded message ----------
> From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
> Date: Mon, Jul 11, 2011 at 10:31 PM
> Subject: trunk FileNotFoundException
> To: swift-user at ci.uchicago.edu
>
>
> Hello,
>
> Using Swift trunk, I am running the SCEC workflow from Communicado using
> ranger, localhost and OSG resources.
>
> One particular app 'mkoffset' which is destined to run on localhost is
> faulting with FileNotFoundException.
>
> The log does give information on its mapping and when it gets 'cleared'.
>
> The config, tc, sites and log files for the run could be found here:
> http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log is 90M, upload size
> exceeded!)
>
> The error stack that I am getting on stdout is:
>
> Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500  Selecting site:390  Stage
> in:16  Active:9  Checking status:1  Finished successfully:36 Failed but can
> retry:3
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>     at
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>     at java.lang.Thread.run(Thread.java:619)
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>     at
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>     at java.lang.Thread.run(Thread.java:619)
> Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500  Selecting site:389  Stage
> in:16  Active:9  Checking status:1  Finished successfully:38 Failed but can
> retry:4
> Execution failed:
>     java.lang.NullPointerException
>     at
> org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
>     at org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
>     at
> org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
>     at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>     at
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>     at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>
>
> Any clues?
>
> Thanks,
> --
> Ketan
>
>
>
>
>
> --
> Ketan
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/0bb19b8b/attachment.html>

From hategan at mcs.anl.gov  Tue Jul 12 20:15:44 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Jul 2011 20:15:44 -0500
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
Message-ID: <1310519744.23008.0.camel@blabla>

Is provider staging on?

On Tue, 2011-07-12 at 14:16 -0500, Ketan Maheshwari wrote:
> Mihael,
> 
> I tried to further investigate the issue and from the logs it seems
> that Swift is trying to execute the mkoffset app before creating a
> jobs/ directory in workdir. Could it be that this is an ordering
> issue. For instance, I see the following line:
> 
> 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2 JOB_START
> jobid=mkoffset-mex8fvck tr=mkoffset arguments=[200.0, 60.0]
> tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
> host=localhost
> 
> but do not see a createdir corresponding to above.
> 
> I have ran this workflow successfully with 0.92.1 so, I am pretty sure
> that it works correctly as far as order of execution is concerned.
> 
> Thanks for any more insights into this.
> 
> Regards,
> Ketan
> 
> ---------- Forwarded message ----------
> From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
> Date: Mon, Jul 11, 2011 at 10:31 PM
> Subject: trunk FileNotFoundException
> To: swift-user at ci.uchicago.edu
> 
> 
> Hello,
> 
> Using Swift trunk, I am running the SCEC workflow from Communicado
> using ranger, localhost and OSG resources.
> 
> One particular app 'mkoffset' which is destined to run on localhost is
> faulting with FileNotFoundException. 
> 
> The log does give information on its mapping and when it gets
> 'cleared'. 
> 
> The config, tc, sites and log files for the run could be found here:
> http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log is 90M, upload
> size exceeded!)
> 
> The error stack that I am getting on stdout is:
> 
> Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500  Selecting site:390
> Stage in:16  Active:9  Checking status:1  Finished successfully:36
> Failed but can retry:3
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> found: /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>     at
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>     at java.lang.Thread.run(Thread.java:619)
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> found: /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>     at
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>     at
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>     at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>     at java.lang.Thread.run(Thread.java:619)
> Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500  Selecting site:389
> Stage in:16  Active:9  Checking status:1  Finished successfully:38
> Failed but can retry:4
> Execution failed:
>     java.lang.NullPointerException
>     at
> org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
>     at
> org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
>     at
> org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
>     at
> org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>     at
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>     at
> org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
>     at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>     at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> 
> 
> Any clues?
> 
> Thanks,
> -- 
> Ketan
> 
> 
> 
> 
> 
> 
> -- 
> Ketan
> 
> 




From hategan at mcs.anl.gov  Tue Jul 12 20:21:31 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Jul 2011 20:21:31 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <79194715-73C7-4A7D-BAAE-DFBDA17FCC87@utexas.edu>
References: <928DD572-EFB5-41CB-ACDA-18464BFD50E1@utexas.edu>
	<1191EA20-7953-4E96-B0B7-7AB4A85756FD@utexas.edu>
	<1310454976.19717.2.camel@blabla>
	<D6303CB9-22C9-4ACC-AF81-C6B67807D3C1@utexas.edu>
	<CAE50RW1-MmrDpbqB7kBWhXJ04Jy4U2UJiLgWs=K=Ummc6j8yYw@mail.gmail.com>
	<79194715-73C7-4A7D-BAAE-DFBDA17FCC87@utexas.edu>
Message-ID: <1310520091.23008.3.camel@blabla>

gsiftp://host:port/path/somefile.txt means a relative path to
somefile.txt (i.e. typically /home/you/path/somefile.txt).

You probably want an absolute path, in which case you need two slashes
after the host[port]:
gsiftp://host:port//gpfs/...

You can look up the URL/URI RFCs for why that is (well, I'm not sure
they really explain why).

On Tue, 2011-07-12 at 17:14 -0500, Jonathan Monette wrote:
> Thanks.  I just verified that something maybe wrong with the path.  I
> could not globus-url-copy that URI so I will continue to investigate
> 
> On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> 
> > i've used this successfully in the past:
> > 
> > file mybrain<single_file_mapper;
> > file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> > 
> > it's been a while though...haven't tested with the latest swift...
> > 
> > On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette
> > <jonmon at utexas.edu> wrote:
> >         How does this syntax look when mapping with the GSIURI?  How
> >         does it work?  I am still getting errors when trying to map
> >         a file with a GSIURI.  It says file not found.  Maybe I have
> >         a different understanding on how it works.
> >         
> >         The files and the logs are in ~jonmon/run.0001 on the ci
> >         machines.
> >         
> >         
> >         On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:
> >         
> >         > You have a colon after the host name but no port. Either
> >         remove the
> >         > colon or put a number after it.
> >         >
> >         > On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
> >         >> I actually meant to send this to swift-devel.
> >         >>
> >         >>
> >         >> Here is the background to the problem.  I have data on
> >         PADS, I am
> >         >> executing Swift on a VM, and I want to use OSG to compute
> >         with the
> >         >> data.  Before Mike left for vacation he said that you can
> >         map data in
> >         >> Swift using the GSIURI scheme but he did not tell me
> >         how.  What I did
> >         >> below is
> >         >>
> >         >>
> >         >> file
> >         >>
> >         data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> >         >>
> >         >>
> >         >> This does not seem to work in release 0.92.1 as the error
> >         below
> >         >> shows.  How do you map data in Swift using the GSIURI
> >         scheme?
> >         >>
> >         >> Begin forwarded message:
> >         >>
> >         >>> From: Jonathan Monette <jonmon at utexas.edu>
> >         >>>
> >         >>> Date: July 11, 2011 2:30:34 PM CDT
> >         >>>
> >         >>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> >         >>>
> >         >>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> >         >>>
> >         >>> Subject: NumberFormatException
> >         >>>
> >         >>>
> >         >>> Mihael,
> >         >>>  I am getting this error using release 0.92.1.
> >         >>>
> >         >>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
> >         >>> id=run:20110711-1924-c944yl9c
> >         >>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext
> >         vdl:new @
> >         >>> script.kml, line: 69: java.lang.RuntimeException:
> >         >>> java.lang.NumberFormatException: For input string: ""
> >         >>> java.lang.RuntimeException:
> >         java.lang.NumberFormatException: For
> >         >>> input string: ""
> >         >>> Caused by: java.lang.RuntimeException:
> >         >>> java.lang.NumberFormatException: For input string: ""
> >         >>> at
> >         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> >         >>> at
> >         >>>
> >         org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> >         >>> at
> >         >>>
> >         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> >         >>> at
> >         >>>
> >         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> >         >>> at
> >         >>>
> >         org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> >         >>> at
> >         org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> >         >>> at
> >         org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> >         >>> at
> >         org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >         >>> at
> >         org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >         >>> at
> >         >>>
> >         org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> >         >>> at
> >         edu.emory.mathcs.backport.java.util.concurrent.Executors
> >         >>> $RunnableAdapter.call(Executors.java:431)
> >         >>> at
> >         >>>
> >         edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >         >>> at
> >         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >         >>> $Worker.runTask(ThreadPoolExecutor.java:643)
> >         >>> at
> >         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >         >>> $Worker.run(ThreadPoolExecutor.java:668)
> >         >>> at java.lang.Thread.run(Thread.java:636)
> >         >>> Caused by: java.lang.NumberFormatException: For input
> >         string: ""
> >         >>> at
> >         >>>
> >         java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >         >>> at java.lang.Integer.parseInt(Integer.java:493)
> >         >>> at java.lang.Integer.parseInt(Integer.java:514)
> >         >>> at
> >         >>>
> >         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> >         >>> at
> >         >>>
> >         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> >         >>> at
> >         org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> >         >>> at
> >         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> >         >>> ... 63 more
> >         >>>
> >         >>> The files needed for this run are located in
> >         ~jonmon/run.0000 on the
> >         >>> ci machines.
> >         >>
> >         >>
> >         >
> >         >
> >         > _______________________________________________
> >         > Swift-devel mailing list
> >         > Swift-devel at ci.uchicago.edu
> >         >
> >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >         
> >         _______________________________________________
> >         Swift-devel mailing list
> >         Swift-devel at ci.uchicago.edu
> >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >         
> > 
> > 
> > 
> > -- 
> > Sarah Kenny
> > Programmer
> > University of Chicago, Computation Institute
> > University of California Irvine, Dept. of Neurology
> > 773-818-8300
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketancmaheshwari at gmail.com  Tue Jul 12 20:26:29 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 12 Jul 2011 20:26:29 -0500
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <1310519744.23008.0.camel@blabla>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
	<1310519744.23008.0.camel@blabla>
Message-ID: <CAMUuvioMHZYAqKvMRqv=t=tMchWhgmFUzMiPGxuMY-L0HXkr7A@mail.gmail.com>

No, provider staging is not on.

I am using local execution provider:

<pool handle="localhost">
    <filesystem provider="local" />
    <execution provider="local" />
    <workdirectory>/var/tmp/swift.workdir</workdirectory>
    <profile namespace="karajan" key="jobThrottle">0.20</profile>
</pool>



On Tue, Jul 12, 2011 at 8:15 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Is provider staging on?
>
> On Tue, 2011-07-12 at 14:16 -0500, Ketan Maheshwari wrote:
> > Mihael,
> >
> > I tried to further investigate the issue and from the logs it seems
> > that Swift is trying to execute the mkoffset app before creating a
> > jobs/ directory in workdir. Could it be that this is an ordering
> > issue. For instance, I see the following line:
> >
> > 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2 JOB_START
> > jobid=mkoffset-mex8fvck tr=mkoffset arguments=[200.0, 60.0]
> > tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
> > host=localhost
> >
> > but do not see a createdir corresponding to above.
> >
> > I have ran this workflow successfully with 0.92.1 so, I am pretty sure
> > that it works correctly as far as order of execution is concerned.
> >
> > Thanks for any more insights into this.
> >
> > Regards,
> > Ketan
> >
> > ---------- Forwarded message ----------
> > From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
> > Date: Mon, Jul 11, 2011 at 10:31 PM
> > Subject: trunk FileNotFoundException
> > To: swift-user at ci.uchicago.edu
> >
> >
> > Hello,
> >
> > Using Swift trunk, I am running the SCEC workflow from Communicado
> > using ranger, localhost and OSG resources.
> >
> > One particular app 'mkoffset' which is destined to run on localhost is
> > faulting with FileNotFoundException.
> >
> > The log does give information on its mapping and when it gets
> > 'cleared'.
> >
> > The config, tc, sites and log files for the run could be found here:
> > http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log is 90M, upload
> > size exceeded!)
> >
> > The error stack that I am getting on stdout is:
> >
> > Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500  Selecting site:390
> > Stage in:16  Active:9  Checking status:1  Finished successfully:36
> > Failed but can retry:3
> > org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> > found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
> >     at
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
> >     at
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
> >     at
> >
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
> >     at java.lang.Thread.run(Thread.java:619)
> > org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> > found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
> >     at
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
> >     at
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
> >     at
> >
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
> >     at
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
> >     at java.lang.Thread.run(Thread.java:619)
> > Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500  Selecting site:389
> > Stage in:16  Active:9  Checking status:1  Finished successfully:38
> > Failed but can retry:4
> > Execution failed:
> >     java.lang.NullPointerException
> >     at
> >
> org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
> >     at
> > org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
> >     at
> > org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >     at
> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >     at
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >
> >
> > Any clues?
> >
> > Thanks,
> > --
> > Ketan
> >
> >
> >
> >
> >
> >
> > --
> > Ketan
> >
> >
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/421d7c8c/attachment.html>

From glen842 at uchicago.edu  Tue Jul 12 20:29:03 2011
From: glen842 at uchicago.edu (Glen Hocky)
Date: Tue, 12 Jul 2011 21:29:03 -0400
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <CAMUuvioMHZYAqKvMRqv=t=tMchWhgmFUzMiPGxuMY-L0HXkr7A@mail.gmail.com>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
	<1310519744.23008.0.camel@blabla>
	<CAMUuvioMHZYAqKvMRqv=t=tMchWhgmFUzMiPGxuMY-L0HXkr7A@mail.gmail.com>
Message-ID: <CAGMfakm++mdpLUyt9=2_R=aU8JsoFbMuKwB6Y0sZNejLs+qLjg@mail.gmail.com>

By the way, I also see this error, w/ pbs+coasters and on osg w/ condor, no
provider staging

On Tue, Jul 12, 2011 at 9:26 PM, Ketan Maheshwari <
ketancmaheshwari at gmail.com> wrote:

> No, provider staging is not on.
>
> I am using local execution provider:
>
> <pool handle="localhost">
>     <filesystem provider="local" />
>     <execution provider="local" />
>     <workdirectory>/var/tmp/swift.workdir</workdirectory>
>     <profile namespace="karajan" key="jobThrottle">0.20</profile>
> </pool>
>
>
>
>
> On Tue, Jul 12, 2011 at 8:15 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
>
>> Is provider staging on?
>>
>> On Tue, 2011-07-12 at 14:16 -0500, Ketan Maheshwari wrote:
>> > Mihael,
>> >
>> > I tried to further investigate the issue and from the logs it seems
>> > that Swift is trying to execute the mkoffset app before creating a
>> > jobs/ directory in workdir. Could it be that this is an ordering
>> > issue. For instance, I see the following line:
>> >
>> > 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2 JOB_START
>> > jobid=mkoffset-mex8fvck tr=mkoffset arguments=[200.0, 60.0]
>> > tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
>> > host=localhost
>> >
>> > but do not see a createdir corresponding to above.
>> >
>> > I have ran this workflow successfully with 0.92.1 so, I am pretty sure
>> > that it works correctly as far as order of execution is concerned.
>> >
>> > Thanks for any more insights into this.
>> >
>> > Regards,
>> > Ketan
>> >
>> > ---------- Forwarded message ----------
>> > From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
>> > Date: Mon, Jul 11, 2011 at 10:31 PM
>> > Subject: trunk FileNotFoundException
>> > To: swift-user at ci.uchicago.edu
>> >
>> >
>> > Hello,
>> >
>> > Using Swift trunk, I am running the SCEC workflow from Communicado
>> > using ranger, localhost and OSG resources.
>> >
>> > One particular app 'mkoffset' which is destined to run on localhost is
>> > faulting with FileNotFoundException.
>> >
>> > The log does give information on its mapping and when it gets
>> > 'cleared'.
>> >
>> > The config, tc, sites and log files for the run could be found here:
>> > http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log is 90M, upload
>> > size exceeded!)
>> >
>> > The error stack that I am getting on stdout is:
>> >
>> > Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500  Selecting site:390
>> > Stage in:16  Active:9  Checking status:1  Finished successfully:36
>> > Failed but can retry:3
>> > org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
>> > found:
>> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>> >     at java.lang.Thread.run(Thread.java:619)
>> > org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
>> > found:
>> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>> >     at
>> >
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>> >     at java.lang.Thread.run(Thread.java:619)
>> > Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500  Selecting site:389
>> > Stage in:16  Active:9  Checking status:1  Finished successfully:38
>> > Failed but can retry:4
>> > Execution failed:
>> >     java.lang.NullPointerException
>> >     at
>> >
>> org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
>> >     at
>> > org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
>> >     at
>> >
>> org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
>> >     at
>> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >     at
>> > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>> >     at
>> >
>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >
>> >
>> > Any clues?
>> >
>> > Thanks,
>> > --
>> > Ketan
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Ketan
>> >
>> >
>>
>>
>>
>
>
> --
> Ketan
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/904313c0/attachment.html>

From hategan at mcs.anl.gov  Tue Jul 12 20:43:38 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 12 Jul 2011 20:43:38 -0500
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <CAGMfakm++mdpLUyt9=2_R=aU8JsoFbMuKwB6Y0sZNejLs+qLjg@mail.gmail.com>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
	<1310519744.23008.0.camel@blabla>
	<CAMUuvioMHZYAqKvMRqv=t=tMchWhgmFUzMiPGxuMY-L0HXkr7A@mail.gmail.com>
	<CAGMfakm++mdpLUyt9=2_R=aU8JsoFbMuKwB6Y0sZNejLs+qLjg@mail.gmail.com>
Message-ID: <1310521418.23008.5.camel@blabla>

The NPE seems legit, and in quite the right spot considering the recent
commits. But I was trying to figure out the thing with the workdir
creation.

In any event, does any of you have a simple test case for the NPE?

On Tue, 2011-07-12 at 21:29 -0400, Glen Hocky wrote:
> By the way, I also see this error, w/ pbs+coasters and on osg w/
> condor, no provider staging
> 
> On Tue, Jul 12, 2011 at 9:26 PM, Ketan Maheshwari
> <ketancmaheshwari at gmail.com> wrote:
>         No, provider staging is not on. 
>         
>         I am using local execution provider:
>         
>         <pool handle="localhost">
>             <filesystem provider="local" />
>             <execution provider="local" />
>             <workdirectory>/var/tmp/swift.workdir</workdirectory>
>             <profile namespace="karajan"
>         key="jobThrottle">0.20</profile>
>         </pool>
>         
>         
>         
>         
>         
>         On Tue, Jul 12, 2011 at 8:15 PM, Mihael Hategan
>         <hategan at mcs.anl.gov> wrote:
>                 Is provider staging on?
>                 
>                 
>                 On Tue, 2011-07-12 at 14:16 -0500, Ketan Maheshwari
>                 wrote:
>                 > Mihael,
>                 >
>                 > I tried to further investigate the issue and from
>                 the logs it seems
>                 > that Swift is trying to execute the mkoffset app
>                 before creating a
>                 > jobs/ directory in workdir. Could it be that this is
>                 an ordering
>                 > issue. For instance, I see the following line:
>                 >
>                 > 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2
>                 JOB_START
>                 > jobid=mkoffset-mex8fvck tr=mkoffset
>                 arguments=[200.0, 60.0]
>                 >
>                 tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
>                 > host=localhost
>                 >
>                 > but do not see a createdir corresponding to above.
>                 >
>                 > I have ran this workflow successfully with 0.92.1
>                 so, I am pretty sure
>                 > that it works correctly as far as order of execution
>                 is concerned.
>                 >
>                 > Thanks for any more insights into this.
>                 >
>                 > Regards,
>                 > Ketan
>                 >
>                 > ---------- Forwarded message ----------
>                 > From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
>                 > Date: Mon, Jul 11, 2011 at 10:31 PM
>                 > Subject: trunk FileNotFoundException
>                 > To: swift-user at ci.uchicago.edu
>                 >
>                 >
>                 > Hello,
>                 >
>                 > Using Swift trunk, I am running the SCEC workflow
>                 from Communicado
>                 > using ranger, localhost and OSG resources.
>                 >
>                 > One particular app 'mkoffset' which is destined to
>                 run on localhost is
>                 > faulting with FileNotFoundException.
>                 >
>                 > The log does give information on its mapping and
>                 when it gets
>                 > 'cleared'.
>                 >
>                 > The config, tc, sites and log files for the run
>                 could be found here:
>                 > http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log
>                 is 90M, upload
>                 > size exceeded!)
>                 >
>                 > The error stack that I am getting on stdout is:
>                 >
>                 > Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500
>                  Selecting site:390
>                 > Stage in:16  Active:9  Checking status:1  Finished
>                 successfully:36
>                 > Failed but can retry:3
>                 >
>                 org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
>                 >
>                 found: /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>                 >     at java.lang.Thread.run(Thread.java:619)
>                 >
>                 org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
>                 >
>                 found: /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
>                 >     at
>                 >
>                 org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
>                 >     at java.lang.Thread.run(Thread.java:619)
>                 > Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500
>                  Selecting site:389
>                 > Stage in:16  Active:9  Checking status:1  Finished
>                 successfully:38
>                 > Failed but can retry:4
>                 > Execution failed:
>                 >     java.lang.NullPointerException
>                 >     at
>                 >
>                 org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
>                 >     at
>                 >
>                 org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
>                 >     at
>                 >
>                 org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
>                 >     at
>                 >
>                 org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>                 >     at
>                 >
>                 org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
>                 >     at
>                 >
>                 org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>                 >
>                 >
>                 > Any clues?
>                 >
>                 > Thanks,
>                 > --
>                 > Ketan
>                 >
>                 >
>                 >
>                 >
>                 >
>                 >
>                 > --
>                 > Ketan
>                 >
>                 >
>                 
>                 
>                 
>         
>         
>         
>         
>         -- 
>         Ketan
>         
>         
>         
>         _______________________________________________
>         Swift-devel mailing list
>         Swift-devel at ci.uchicago.edu
>         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
> 




From hockyg at uchicago.edu  Tue Jul 12 21:37:42 2011
From: hockyg at uchicago.edu (Glen Hocky)
Date: Tue, 12 Jul 2011 22:37:42 -0400
Subject: [Swift-devel] stale nfs problem
Message-ID: <CAGMfakk+rzuBHxPp5BNkSxKypN3NZgLHBRnQE=A12LcYPLy2+Q@mail.gmail.com>

Hey everyone,
this isn't quite a bug but more of a feature selection. I had some jobs that
I started with swiftwork on /gpfs/pads before they turned it off for
maintenance. The jobs in the queue were put on hold and then restarted after
pads was back. Jobs ran or were running, but when they started to finish, I
got the error

 Progress:  time: Tue, 12 Jul 2011 20:31:01 -0600  Active:382  Checking
> status:1  Finished successfully:1
> Failed to sync restart log
> java.io.IOException: Stale NFS file handle
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:277)
>         at
> sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:355)
>         at
> sun.nio.cs.StreamEncoder$CharsetSE.implWrite(StreamEncoder.java:416)
>         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:159)
>         at
> java.io.OutputStreamWriter.emptyBuffer(OutputStreamWriter.java:290)
>         at java.io.OutputStreamWriter.flush(OutputStreamWriter.java)
>         at
> org.globus.cog.karajan.workflow.nodes.restartLog.FlushableLockedFileWriter.actualFlush(FlushableLockedFileWriter.java:47)
>         at
> org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.doFlush(SyncThread.java:59)
>         at
> org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.run(SyncThread.java:50)
> Progress:  time: Tue, 12 Jul 2011 20:31:06 -0600  Active:382  Finished
> successfully:2
> Progress:  time: Tue, 12 Jul 2011 20:31:23 -0600  Active:381  Checking
> status:1  Finished successfully:2
> Failed to close log file
> java.io.IOException: Stale NFS file handle


Perhaps in future versions it would be possible to catch this exception and
make one try at rectifying the situation

-Glen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110712/20c11ffd/attachment.html>

From iraicu at cs.iit.edu  Wed Jul 13 01:22:29 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Wed, 13 Jul 2011 01:22:29 -0500
Subject: [Swift-devel] CFP: 12th IEEE/ACM Int. Symp. on Cluster,
 Grid and Cloud, Computing (CCGrid) 2012
Message-ID: <4E1D39A5.4090001@cs.iit.edu>

12th IEEE/ACM International Symposium on Cluster, Grid and Cloud
Computing (CCGrid 2012) Ottawa, Canada

May 13-16, 2012

http://www.cloudbus.org/ccgrid2012


CALL FOR PAPERS

Rapid advances in processing, communication and systems/middleware
technologies are leading to new paradigms and platforms for computing,
ranging from computing Clusters to widely distributed Grid and
emerging Clouds. CCGrid is a series of very successful conferences,
sponsored by the IEEE Computer Society Technical Committee on Scalable
Computing (TCSC) and ACM, with the overarching goal of bringing
together international researchers, developers, and users and to
provide an international forum to present leading research activities
and results on a broad range of topics related to these platforms and
paradigms and their applications. The conference features keynotes,
technical presentations, posters and research demos, workshops,
tutorials, as well as the SCALE challenges featuring live
demonstrations.

In 2012, CCGrid will come to Canada for the first time and will be
held in Ottawa, the capital city. CCGrid 2012 will have a focus on
important and immediate issues that are significantly influencing all
aspects of cluster, cloud and grid computing. Topics of interest
include, but are not limited to:

  * Applications and Experiences: Applications to real and complex
    problems in science, engineering, business and society; User
    studies; Experiences with large-scale deployments systems or
    applications.

  * Architecture: System architectures, Design and deployment.

  * Autonomic Computing and Cyberinfrastructure: Self managed behavior,
    models and technologies; Autonomic paradigms and approaches
    (control-based, bio-inspired, emergent, etc.); Bio-inspired
    approaches to management; SLA definition and enforcement;

  * Performance Modeling and Evaluation: Performance models; Monitoring
    and evaluation tools, Analysis of system/application performance;
    Benchmarks and testbeds.

  * Programming Models, Systems, and Fault-Tolerant Computing:
    Programming models for cluster, clouds and grid computing; fault
    tolerant infrastructure and algorithms; systems software to enable
    efficient computing.

  * Multicore and Accelerator-based Computing: Software and application
    techniques to utilize multicore architectures and
    accelerators/heterogeneous computing systems.

  * Scheduling and Resource Management: Techniques to schedule jobs and
    resources on clusters, clouds and grid computing platforms.

  * Cloud Computing: Cloud architectures; Software tools and techniques
    for clouds.


PAPER SUBMISSION

Authors are invited to submit papers electronically. Submitted
manuscripts should be structured as technical papers and may not
exceed 8 letter size (8.5 x 11) pages including figures, tables and
references using the IEEE format for conference proceedings (print
area of 6-1/2 inches (16.51 cm) wide by 8-7/8 inches (22.51 cm) high,
two-column format with columns 3-1/16 inches (7.85 cm) wide with a 3/8
inch (0.81 cm) space between them, single-spaced 10-point Times fully
justified text). Submissions not conforming to these guidelines may be
returned without review. Authors should submit the manuscript in PDF
format and make sure that the file will print on a printer that uses
letter size (8.5 x 11) paper. The official language of the meeting is
English. All manuscripts will be reviewed and will be judged on
correctness, originality, technical strength, significance, quality of
presentation, and interest and relevance to the conference attendees.

Submitted papers must represent original unpublished research that is
not currently under review for any other conference or journal. Papers
not following these guidelines will be rejected without review and
further action may be taken, including (but not limited to)
notifications sent to the heads of the institutions of the authors and
sponsors of the conference. Submissions received after the due date,
exceeding the page limit, or not appropriately structured may not be
considered. Authors may contact the conference chairs for more
information. The proceedings will be published through the IEEE
Computer Society Press, USA and will be made available online through
the IEEE Digital Library.

Submission Link:
https://www.easychair.org/account/signin.cgi?conf=ccgrid2012


CHAIRS

General Chair
  * Shikharesh Majumdar, Carleton University, Canada

Program Committee Co-Chairs
  * Rajkumar Buyya, University of Melbourne, Australia
  * Pavan Balaji, Argonne National Laboratory, USA

Program Committee Vice-chairs
  * Daniel S. Katz (Applications and Experiences)
  * Dhabaleswar K. Panda (Architecture)
  * Manish Parashar (Middleware, Autonomic Computing, and 
Cyberinfrastructure)
  * Ahmad Afsahi (Performance Modeling and Analysis)
  * Xian-He Sun (Performance Measurement and Evaluation)
  * William Gropp (Programming Models, Systems, and Fault-Tolerant 
computing)
  * David Bader (Multicore and Accelerator-based Computing)
  * Thomas Fahringer (Scheduling and Resource Management)
  * Ignacio Martin Llorente and Madhusudhan Govindaraju (Cloud Computing)

Honorary Chair
  * Geoffrey Fox, Indiana University, USA


IMPORTANT DATES

   Papers Due:             25 November 2011
   Notification of Acceptance:     30 January 2012
   Camera Ready Papers Due:     27 February 2012

   Sponsors: IEEE Computer Society (TCSE) & ACM SIGARCH (approval pending)

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================




From wilde at mcs.anl.gov  Wed Jul 13 09:13:25 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 13 Jul 2011 09:13:25 -0500 (CDT)
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <79194715-73C7-4A7D-BAAE-DFBDA17FCC87@utexas.edu>
Message-ID: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>

I was going to suggest that same as Sarah pointed out, suspecting that the short-form of single_file_mapper is not correctly parsing the GSIFTP URI. If thats the case, please file it as a bug, Jon. 


Can you try both forms of the mapper syntax with a known-good URI? 


Thanks, 


- Mike 


----- Original Message -----


Thanks. I just verified that something maybe wrong with the path. I could not globus-url-copy that URI so I will continue to investigate 




On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote: 


i've used this successfully in the past: 

file mybrain<single_file_mapper; file="gsiftp:// calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz ")>; 

it's been a while though...haven't tested with the latest swift... 


On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette < jonmon at utexas.edu > wrote: 


How does this syntax look when mapping with the GSIURI? How does it work? I am still getting errors when trying to map a file with a GSIURI. It says file not found. Maybe I have a different understanding on how it works. 

The files and the logs are in ~jonmon/run.0001 on the ci machines. 




On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote: 

> You have a colon after the host name but no port. Either remove the 
> colon or put a number after it. 
> 
> On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote: 
>> I actually meant to send this to swift-devel. 
>> 
>> 
>> Here is the background to the problem. I have data on PADS, I am 
>> executing Swift on a VM, and I want to use OSG to compute with the 
>> data. Before Mike left for vacation he said that you can map data in 
>> Swift using the GSIURI scheme but he did not tell me how. What I did 
>> below is 
>> 
>> 
>> file 
>> data<" gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt ">; 
>> 
>> 
>> This does not seem to work in release 0.92.1 as the error below 
>> shows. How do you map data in Swift using the GSIURI scheme? 
>> 
>> Begin forwarded message: 
>> 
>>> From: Jonathan Monette < jonmon at utexas.edu > 
>>> 
>>> Date: July 11, 2011 2:30:34 PM CDT 
>>> 
>>> To: swift-devel Devel < swift-devel at ci.uchicago.edu > 
>>> 
>>> Cc: Mihael Hategan Hategan < hategan at mcs.anl.gov > 
>>> 
>>> Subject: NumberFormatException 
>>> 
>>> 
>>> Mihael, 
>>> I am getting this error using release 0.92.1. 
>>> 
>>> 2011-07-11 19:24:44,395+0000 INFO unknown RUNID 
>>> id=run:20110711-1924-c944yl9c 
>>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @ 
>>> script.kml, line: 69: java.lang.RuntimeException: 
>>> java.lang.NumberFormatException: For input string: "" 
>>> java.lang.RuntimeException: java.lang.NumberFormatException: For 
>>> input string: "" 
>>> Caused by: java.lang.RuntimeException: 
>>> java.lang.NumberFormatException: For input string: "" 
>>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109) 
>>> at 
>>> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24) 
>>> at 
>>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97) 
>>> at 
>>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75) 
>>> at 
>>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61) 
>>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37) 
>>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126) 
>>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) 
>>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402) 
>>> at 
>>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173) 
>>> at 
>>> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44) 
>>> at edu.emory.mathcs.backport.java.util.concurrent.Executors 
>>> $RunnableAdapter.call(Executors.java:431) 
>>> at 
>>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166) 
>>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
>>> $Worker.runTask(ThreadPoolExecutor.java:643) 
>>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor 
>>> $Worker.run(ThreadPoolExecutor.java:668) 
>>> at java.lang.Thread.run(Thread.java:636) 
>>> Caused by: java.lang.NumberFormatException: For input string: "" 
>>> at 
>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) 
>>> at java.lang.Integer.parseInt(Integer.java:493) 
>>> at java.lang.Integer.parseInt(Integer.java:514) 
>>> at 
>>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90) 
>>> at 
>>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27) 
>>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84) 
>>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99) 
>>> ... 63 more 
>>> 
>>> The files needed for this run are located in ~jonmon/run.0000 on the 
>>> ci machines. 
>> 
>> 
> 
> 
> _______________________________________________ 
> Swift-devel mailing list 
> Swift-devel at ci.uchicago.edu 
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 

_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Sarah Kenny 
Programmer 
University of Chicago, Computation Institute 
University of California Irvine, Dept. of Neurology 
773-818-8300 

_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 


_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110713/399056a2/attachment.html>

From jonmon at utexas.edu  Wed Jul 13 11:29:58 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Wed, 13 Jul 2011 11:29:58 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>
References: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>
Message-ID: <D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>

I am starting to believe that this maybe a bug.  I have this line in the log.

2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile FILE_STAGE_IN_START file=data.txt srchost=gridftp.pads.ci.uchicago.edu:2811 srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test srcname=data.txt desthost=localhost destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test provider=gsiftp policy=DEFAULT

The desthost is localhost.  That is correct.  But why does the destdir have /gpfs in it?  My localhost is VM.  It does not have access to gpfs.

The files for this run are in ~jonmon/run.0002 on the ci machines.

On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:

> I was going to suggest that same as Sarah pointed out, suspecting that the short-form of single_file_mapper is not correctly parsing the GSIFTP URI.  If thats the case, please file it as a bug, Jon.
> 
> Can you try both forms of the mapper syntax with a known-good URI?
> 
> Thanks,
> 
> - Mike
> 
> 
> Thanks.  I just verified that something maybe wrong with the path.  I could not globus-url-copy that URI so I will continue to investigate
> 
> On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> 
> i've used this successfully in the past:
> 
> file mybrain<single_file_mapper; file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> 
> it's been a while though...haven't tested with the latest swift...
> 
> On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette <jonmon at utexas.edu> wrote:
> How does this syntax look when mapping with the GSIURI?  How does it work?  I am still getting errors when trying to map a file with a GSIURI.  It says file not found.  Maybe I have a different understanding on how it works.
> 
> The files and the logs are in ~jonmon/run.0001 on the ci machines.
> 
> On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:
> 
> > You have a colon after the host name but no port. Either remove the
> > colon or put a number after it.
> >
> > On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
> >> I actually meant to send this to swift-devel.
> >>
> >>
> >> Here is the background to the problem.  I have data on PADS, I am
> >> executing Swift on a VM, and I want to use OSG to compute with the
> >> data.  Before Mike left for vacation he said that you can map data in
> >> Swift using the GSIURI scheme but he did not tell me how.  What I did
> >> below is
> >>
> >>
> >> file
> >> data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> >>
> >>
> >> This does not seem to work in release 0.92.1 as the error below
> >> shows.  How do you map data in Swift using the GSIURI scheme?
> >>
> >> Begin forwarded message:
> >>
> >>> From: Jonathan Monette <jonmon at utexas.edu>
> >>>
> >>> Date: July 11, 2011 2:30:34 PM CDT
> >>>
> >>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
> >>>
> >>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
> >>>
> >>> Subject: NumberFormatException
> >>>
> >>>
> >>> Mihael,
> >>>  I am getting this error using release 0.92.1.
> >>>
> >>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
> >>> id=run:20110711-1924-c944yl9c
> >>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
> >>> script.kml, line: 69: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> java.lang.RuntimeException: java.lang.NumberFormatException: For
> >>> input string: ""
> >>> Caused by: java.lang.RuntimeException:
> >>> java.lang.NumberFormatException: For input string: ""
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> >>> at
> >>> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> >>> at
> >>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> >>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> >>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> >>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> >>> at
> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> >>> at
> >>> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.Executors
> >>> $RunnableAdapter.call(Executors.java:431)
> >>> at
> >>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.runTask(ThreadPoolExecutor.java:643)
> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.run(ThreadPoolExecutor.java:668)
> >>> at java.lang.Thread.run(Thread.java:636)
> >>> Caused by: java.lang.NumberFormatException: For input string: ""
> >>> at
> >>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >>> at java.lang.Integer.parseInt(Integer.java:493)
> >>> at java.lang.Integer.parseInt(Integer.java:514)
> >>> at
> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> >>> at
> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> >>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> >>> ... 63 more
> >>>
> >>> The files needed for this run are located in ~jonmon/run.0000 on the
> >>> ci machines.
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> -- 
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110713/2e5c54db/attachment.html>

From jonmon at utexas.edu  Wed Jul 13 13:00:24 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Wed, 13 Jul 2011 13:00:24 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>
References: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>
	<D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>
Message-ID: <848D1D4C-057D-470A-B699-ABFA05CBE2F5@gmail.com>

Mike mentioned that this path is correct and makes sense.  I was able to use the globus-url-copy from the swift bin directory and make a copy from that GSIURI.  The error that I am receiving is 

Caused by: Exception in getFile
Caused by: org.globus.cog.abstraction.impl.file.IrrecoverableResourceException: Exception in getFile
Caused by: org.globus.ftp.exception.ServerException: Reply wait timeout. (error code 4)
	at org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
	at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
	at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
	at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
	at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
	at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
	at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
	at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
	at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
	at org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
	at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
	at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
	at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
	at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
	at java.lang.Thread.run(Thread.java:636)

I don't know if I mentioned this but this is with the 0.92.1 release.

On Jul 13, 2011, at 11:29 AM, Jonathan Monette wrote:

> I am starting to believe that this maybe a bug.  I have this line in the log.
> 
> 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile FILE_STAGE_IN_START file=data.txt srchost=gridftp.pads.ci.uchicago.edu:2811 srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test srcname=data.txt desthost=localhost destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test provider=gsiftp policy=DEFAULT
> 
> The desthost is localhost.  That is correct.  But why does the destdir have /gpfs in it?  My localhost is VM.  It does not have access to gpfs.
> 
> The files for this run are in ~jonmon/run.0002 on the ci machines.
> 
> On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:
> 
>> I was going to suggest that same as Sarah pointed out, suspecting that the short-form of single_file_mapper is not correctly parsing the GSIFTP URI.  If thats the case, please file it as a bug, Jon.
>> 
>> Can you try both forms of the mapper syntax with a known-good URI?
>> 
>> Thanks,
>> 
>> - Mike
>> 
>> 
>> Thanks.  I just verified that something maybe wrong with the path.  I could not globus-url-copy that URI so I will continue to investigate
>> 
>> On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
>> 
>> i've used this successfully in the past:
>> 
>> file mybrain<single_file_mapper; file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
>> 
>> it's been a while though...haven't tested with the latest swift...
>> 
>> On Tue, Jul 12, 2011 at 1:59 PM, Jonathan Monette <jonmon at utexas.edu> wrote:
>> How does this syntax look when mapping with the GSIURI?  How does it work?  I am still getting errors when trying to map a file with a GSIURI.  It says file not found.  Maybe I have a different understanding on how it works.
>> 
>> The files and the logs are in ~jonmon/run.0001 on the ci machines.
>> 
>> On Jul 12, 2011, at 2:16 AM, Mihael Hategan wrote:
>> 
>> > You have a colon after the host name but no port. Either remove the
>> > colon or put a number after it.
>> >
>> > On Mon, 2011-07-11 at 22:45 -0500, Jonathan Monette wrote:
>> >> I actually meant to send this to swift-devel.
>> >>
>> >>
>> >> Here is the background to the problem.  I have data on PADS, I am
>> >> executing Swift on a VM, and I want to use OSG to compute with the
>> >> data.  Before Mike left for vacation he said that you can map data in
>> >> Swift using the GSIURI scheme but he did not tell me how.  What I did
>> >> below is
>> >>
>> >>
>> >> file
>> >> data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
>> >>
>> >>
>> >> This does not seem to work in release 0.92.1 as the error below
>> >> shows.  How do you map data in Swift using the GSIURI scheme?
>> >>
>> >> Begin forwarded message:
>> >>
>> >>> From: Jonathan Monette <jonmon at utexas.edu>
>> >>>
>> >>> Date: July 11, 2011 2:30:34 PM CDT
>> >>>
>> >>> To: swift-devel Devel <swift-devel at ci.uchicago.edu>
>> >>>
>> >>> Cc: Mihael Hategan Hategan <hategan at mcs.anl.gov>
>> >>>
>> >>> Subject: NumberFormatException
>> >>>
>> >>>
>> >>> Mihael,
>> >>>  I am getting this error using release 0.92.1.
>> >>>
>> >>> 2011-07-11 19:24:44,395+0000 INFO  unknown RUNID
>> >>> id=run:20110711-1924-c944yl9c
>> >>> 2011-07-11 19:24:44,508+0000 DEBUG VDL2ExecutionContext vdl:new @
>> >>> script.kml, line: 69: java.lang.RuntimeException:
>> >>> java.lang.NumberFormatException: For input string: ""
>> >>> java.lang.RuntimeException: java.lang.NumberFormatException: For
>> >>> input string: ""
>> >>> Caused by: java.lang.RuntimeException:
>> >>> java.lang.NumberFormatException: For input string: ""
>> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
>> >>> at
>> >>> org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
>> >>> at
>> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
>> >>> at
>> >>> org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
>> >>> at
>> >>> org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
>> >>> at org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
>> >>> at org.griphyn.vdl.karajan.lib.New.function(New.java:126)
>> >>> at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >>> at org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>> >>> at
>> >>> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
>> >>> at edu.emory.mathcs.backport.java.util.concurrent.Executors
>> >>> $RunnableAdapter.call(Executors.java:431)
>> >>> at
>> >>> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>> >>> $Worker.runTask(ThreadPoolExecutor.java:643)
>> >>> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>> >>> $Worker.run(ThreadPoolExecutor.java:668)
>> >>> at java.lang.Thread.run(Thread.java:636)
>> >>> Caused by: java.lang.NumberFormatException: For input string: ""
>> >>> at
>> >>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> >>> at java.lang.Integer.parseInt(Integer.java:493)
>> >>> at java.lang.Integer.parseInt(Integer.java:514)
>> >>> at
>> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
>> >>> at
>> >>> org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
>> >>> at org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
>> >>> at org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
>> >>> ... 63 more
>> >>>
>> >>> The files needed for this run are located in ~jonmon/run.0000 on the
>> >>> ci machines.
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> 
>> -- 
>> Sarah Kenny
>> Programmer
>> University of Chicago, Computation Institute
>> University of California Irvine, Dept. of Neurology
>> 773-818-8300
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> 
>> -- 
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110713/287065f5/attachment.html>

From wozniak at mcs.anl.gov  Wed Jul 13 15:39:26 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 13 Jul 2011 15:39:26 -0500 (CDT)
Subject: [Swift-devel] Call today
Message-ID: <alpine.DEB.2.00.1107131539120.1980@wozniak-desktop.mcs.anl.gov>


I'll be on Skype for the meeting today.

-- 
Justin M Wozniak


From hategan at mcs.anl.gov  Wed Jul 13 18:55:01 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Jul 2011 18:55:01 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>
References: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>
	<D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>
Message-ID: <1310601301.2210.6.camel@blabla>

On Wed, 2011-07-13 at 11:29 -0500, Jonathan Monette wrote:
> I am starting to believe that this maybe a bug.  I have this line in
> the log.
> 
> 
> 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile
> FILE_STAGE_IN_START file=data.txt
> srchost=gridftp.pads.ci.uchicago.edu:2811
> srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> srcname=data.txt desthost=localhost
> destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test provider=gsiftp policy=DEFAULT
> 
> 
> The desthost is localhost.  That is correct.  But why does the destdir
> have /gpfs in it?  My localhost is VM.  It does not have access to
> gpfs.

It makes (some) sense because this is a simple way to ensure that no two
files on a remote filesystem will have the same path inside the job
directory if they have different paths on the source side. This is
mostly true if the source side is on a single filesystem (which is what
the initial version of swift did). With more than one (i.e. cat(a, b)
where a = host1/p/f.txt, b = host2/p/f.txt) it's not quite so (and we
should probably fix that). In any event, it's what it's supposed to be
in the current implementation.





From hategan at mcs.anl.gov  Wed Jul 13 18:57:54 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Jul 2011 18:57:54 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <848D1D4C-057D-470A-B699-ABFA05CBE2F5@gmail.com>
References: <324758805.90555.1310566405189.JavaMail.root@zimbra.anl.gov>
	<D00EDEE1-F43F-4766-A8FB-00D1783FB1E2@utexas.edu>
	<848D1D4C-057D-470A-B699-ABFA05CBE2F5@gmail.com>
Message-ID: <1310601474.2210.8.camel@blabla>

The full log will help here, but the "Reply wait timeout" exception is
not necessarily an indication of a bad path (though I'm not saying it's
not).

In the meantime, can you check that you have GLOBUS_HOSTNAME set
properly and that your client machine is not behind a firewall?

On Wed, 2011-07-13 at 13:00 -0500, Jonathan Monette wrote:
> Mike mentioned that this path is correct and makes sense.  I was able
> to use the globus-url-copy from the swift bin directory and make a
> copy from that GSIURI.  The error that I am receiving is 
> 
> 
> Caused by: Exception in getFile
> Caused by:
> org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> Exception in getFile
> Caused by: org.globus.ftp.exception.ServerException: Reply wait
> timeout. (error code 4)
> at
> org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
> at
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
> at
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> at
> org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> at
> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> at
> org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> at
> org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> at
> org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> at
> org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> at
> org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> at
> org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
> at
> org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> at
> org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> at edu.emory.mathcs.backport.java.util.concurrent.Executors
> $RunnableAdapter.call(Executors.java:431)
> at
> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:643)
> at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:668)
> at java.lang.Thread.run(Thread.java:636)
> 
> 
> I don't know if I mentioned this but this is with the 0.92.1 release.
> 
> 
> On Jul 13, 2011, at 11:29 AM, Jonathan Monette wrote:
> 
> > I am starting to believe that this maybe a bug.  I have this line in
> > the log.
> > 
> > 
> > 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile
> > FILE_STAGE_IN_START file=data.txt
> > srchost=gridftp.pads.ci.uchicago.edu:2811
> > srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > srcname=data.txt desthost=localhost
> > destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test provider=gsiftp policy=DEFAULT
> > 
> > 
> > The desthost is localhost.  That is correct.  But why does the
> > destdir have /gpfs in it?  My localhost is VM.  It does not have
> > access to gpfs.
> > 
> > 
> > The files for this run are in ~jonmon/run.0002 on the ci machines.
> > 
> > 
> > On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:
> > 
> > > I was going to suggest that same as Sarah pointed out, suspecting
> > > that the short-form of single_file_mapper is not correctly parsing
> > > the GSIFTP URI.  If thats the case, please file it as a bug, Jon.
> > > 
> > > 
> > > Can you try both forms of the mapper syntax with a known-good URI?
> > > 
> > > 
> > > Thanks,
> > > 
> > > 
> > > - Mike
> > > 
> > > 
> > > 
> > > __________________________________________________________________
> > >         Thanks.  I just verified that something maybe wrong with
> > >         the path.  I could not globus-url-copy that URI so I will
> > >         continue to investigate
> > >         
> > >         On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> > >         
> > >                 i've used this successfully in the past:
> > >                 
> > >                 file mybrain<single_file_mapper;
> > >                 file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> > >                 
> > >                 it's been a while though...haven't tested with the
> > >                 latest swift...
> > >                 
> > >                 On Tue, Jul 12, 2011 at 1:59 PM, Jonathan
> > >                 Monette <jonmon at utexas.edu> wrote:
> > >                         How does this syntax look when mapping
> > >                         with the GSIURI?  How does it work?  I am
> > >                         still getting errors when trying to map a
> > >                         file with a GSIURI.  It says file not
> > >                         found.  Maybe I have a different
> > >                         understanding on how it works.
> > >                         
> > >                         The files and the logs are in
> > >                         ~jonmon/run.0001 on the ci machines.
> > >                         
> > >                         
> > >                         On Jul 12, 2011, at 2:16 AM, Mihael
> > >                         Hategan wrote:
> > >                         
> > >                         > You have a colon after the host name but
> > >                         no port. Either remove the
> > >                         > colon or put a number after it.
> > >                         >
> > >                         > On Mon, 2011-07-11 at 22:45 -0500,
> > >                         Jonathan Monette wrote:
> > >                         >> I actually meant to send this to
> > >                         swift-devel.
> > >                         >>
> > >                         >>
> > >                         >> Here is the background to the problem.
> > >                          I have data on PADS, I am
> > >                         >> executing Swift on a VM, and I want to
> > >                         use OSG to compute with the
> > >                         >> data.  Before Mike left for vacation he
> > >                         said that you can map data in
> > >                         >> Swift using the GSIURI scheme but he
> > >                         did not tell me how.  What I did
> > >                         >> below is
> > >                         >>
> > >                         >>
> > >                         >> file
> > >                         >>
> > >                         data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> > >                         >>
> > >                         >>
> > >                         >> This does not seem to work in release
> > >                         0.92.1 as the error below
> > >                         >> shows.  How do you map data in Swift
> > >                         using the GSIURI scheme?
> > >                         >>
> > >                         >> Begin forwarded message:
> > >                         >>
> > >                         >>> From: Jonathan Monette
> > >                         <jonmon at utexas.edu>
> > >                         >>>
> > >                         >>> Date: July 11, 2011 2:30:34 PM CDT
> > >                         >>>
> > >                         >>> To: swift-devel Devel
> > >                         <swift-devel at ci.uchicago.edu>
> > >                         >>>
> > >                         >>> Cc: Mihael Hategan Hategan
> > >                         <hategan at mcs.anl.gov>
> > >                         >>>
> > >                         >>> Subject: NumberFormatException
> > >                         >>>
> > >                         >>>
> > >                         >>> Mihael,
> > >                         >>>  I am getting this error using release
> > >                         0.92.1.
> > >                         >>>
> > >                         >>> 2011-07-11 19:24:44,395+0000 INFO
> > >                          unknown RUNID
> > >                         >>> id=run:20110711-1924-c944yl9c
> > >                         >>> 2011-07-11 19:24:44,508+0000 DEBUG
> > >                         VDL2ExecutionContext vdl:new @
> > >                         >>> script.kml, line: 69:
> > >                         java.lang.RuntimeException:
> > >                         >>> java.lang.NumberFormatException: For
> > >                         input string: ""
> > >                         >>> java.lang.RuntimeException:
> > >                         java.lang.NumberFormatException: For
> > >                         >>> input string: ""
> > >                         >>> Caused by: java.lang.RuntimeException:
> > >                         >>> java.lang.NumberFormatException: For
> > >                         input string: ""
> > >                         >>> at
> > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> > >                         >>> at
> > >                         >>>
> > >                         org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> > >                         >>> at
> > >                         >>>
> > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> > >                         >>> at
> > >                         >>>
> > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> > >                         >>> at
> > >                         >>>
> > >                         org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> > >                         >>> at
> > >                         org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> > >                         >>> at
> > >                         org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> > >                         >>> at
> > >                         org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > >                         >>> at
> > >                         org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > >                         >>> at
> > >                         edu.emory.mathcs.backport.java.util.concurrent.Executors
> > >                         >>>
> > >                         $RunnableAdapter.call(Executors.java:431)
> > >                         >>> at
> > >                         >>>
> > >                         edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > >                         >>> at
> > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > >                         >>>
> > >                         $Worker.runTask(ThreadPoolExecutor.java:643)
> > >                         >>> at
> > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > >                         >>>
> > >                         $Worker.run(ThreadPoolExecutor.java:668)
> > >                         >>> at
> > >                         java.lang.Thread.run(Thread.java:636)
> > >                         >>> Caused by:
> > >                         java.lang.NumberFormatException: For input
> > >                         string: ""
> > >                         >>> at
> > >                         >>>
> > >                         java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > >                         >>> at
> > >                         java.lang.Integer.parseInt(Integer.java:493)
> > >                         >>> at
> > >                         java.lang.Integer.parseInt(Integer.java:514)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> > >                         >>> at
> > >                         >>>
> > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> > >                         >>> at
> > >                         org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> > >                         >>> at
> > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> > >                         >>> ... 63 more
> > >                         >>>
> > >                         >>> The files needed for this run are
> > >                         located in ~jonmon/run.0000 on the
> > >                         >>> ci machines.
> > >                         >>
> > >                         >>
> > >                         >
> > >                         >
> > >                         >
> > >                         _______________________________________________
> > >                         > Swift-devel mailing list
> > >                         > Swift-devel at ci.uchicago.edu
> > >                         > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >                         
> > >                         _______________________________________________
> > >                         Swift-devel mailing list
> > >                         Swift-devel at ci.uchicago.edu
> > >                         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >                         
> > >                 
> > >                 
> > >                 
> > >                 -- 
> > >                 Sarah Kenny
> > >                 Programmer
> > >                 University of Chicago, Computation Institute
> > >                 University of California Irvine, Dept. of
> > >                 Neurology
> > >                 773-818-8300
> > >                 
> > >                 _______________________________________________
> > >                 Swift-devel mailing list
> > >                 Swift-devel at ci.uchicago.edu
> > >                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >         
> > >         
> > >         
> > >         _______________________________________________
> > >         Swift-devel mailing list
> > >         Swift-devel at ci.uchicago.edu
> > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > 
> > > 
> > > 
> > > -- 
> > > Michael Wilde
> > > Computation Institute, University of Chicago
> > > Mathematics and Computer Science Division
> > > Argonne National Laboratory
> > > 
> > > 
> > 
> > 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Wed Jul 13 18:59:59 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Jul 2011 18:59:59 -0500
Subject: [Swift-devel] Call today
In-Reply-To: <alpine.DEB.2.00.1107131539120.1980@wozniak-desktop.mcs.anl.gov>
References: <alpine.DEB.2.00.1107131539120.1980@wozniak-desktop.mcs.anl.gov>
Message-ID: <1310601599.2210.10.camel@blabla>

Sorry. I was on planes heading towards Davis earlier today. I should
have sent a message earlier, but my aunt died and I went to her funeral
yesterday.

Mihael

On Wed, 2011-07-13 at 15:39 -0500, Justin M Wozniak wrote:
> I'll be on Skype for the meeting today.
> 




From hategan at mcs.anl.gov  Wed Jul 13 19:02:41 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Jul 2011 19:02:41 -0500
Subject: [Swift-devel] Call today
In-Reply-To: <1310601599.2210.10.camel@blabla>
References: <alpine.DEB.2.00.1107131539120.1980@wozniak-desktop.mcs.anl.gov>
	<1310601599.2210.10.camel@blabla>
Message-ID: <1310601761.2210.13.camel@blabla>

On Wed, 2011-07-13 at 18:59 -0500, Mihael Hategan wrote:
> Sorry. I was on planes heading towards Davis earlier today. I should
> have sent a message earlier, but my aunt died and I went to her funeral
> yesterday.

And, well, that was a useless message...

Can we do a phone call tomorrow? One regarding the last few commits is
overdue I think.

Mihael



From wilde at mcs.anl.gov  Wed Jul 13 19:09:21 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 13 Jul 2011 19:09:21 -0500 (CDT)
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <1310601474.2210.8.camel@blabla>
Message-ID: <1700525737.93869.1310602161459.JavaMail.root@zimbra.anl.gov>

If this is the same error I think it is, its likely due to the fact that Swift was trying to use 3rd party transfer with the Swift client on an EC2 VM which was unreachable from the PADS GridFTP server.

It raised some interesting issues on how Swift tried to do transfers from a file mapped to a gsiftp:// URI and a local execution site which I think are still unresolved but were the likely cause of this error.

I think Jon is reworking this test case to use only remote GridFTp-enabled sites outside of EC2 as the execution sites, and the correct DNS or IP name for inbound connections back to Swift.

- Mike


----- Original Message -----
> The full log will help here, but the "Reply wait timeout" exception is
> not necessarily an indication of a bad path (though I'm not saying
> it's
> not).
> 
> In the meantime, can you check that you have GLOBUS_HOSTNAME set
> properly and that your client machine is not behind a firewall?
> 
> On Wed, 2011-07-13 at 13:00 -0500, Jonathan Monette wrote:
> > Mike mentioned that this path is correct and makes sense. I was able
> > to use the globus-url-copy from the swift bin directory and make a
> > copy from that GSIURI. The error that I am receiving is
> >
> >
> > Caused by: Exception in getFile
> > Caused by:
> > org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> > Exception in getFile
> > Caused by: org.globus.ftp.exception.ServerException: Reply wait
> > timeout. (error code 4)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
> > at
> > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > at
> > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > at
> > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > at
> > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > at
> > org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
> > at
> > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > at
> > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > at edu.emory.mathcs.backport.java.util.concurrent.Executors
> > $RunnableAdapter.call(Executors.java:431)
> > at
> > edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:643)
> > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:668)
> > at java.lang.Thread.run(Thread.java:636)
> >
> >
> > I don't know if I mentioned this but this is with the 0.92.1
> > release.
> >
> >
> > On Jul 13, 2011, at 11:29 AM, Jonathan Monette wrote:
> >
> > > I am starting to believe that this maybe a bug. I have this line
> > > in
> > > the log.
> > >
> > >
> > > 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile
> > > FILE_STAGE_IN_START file=data.txt
> > > srchost=gridftp.pads.ci.uchicago.edu:2811
> > > srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > srcname=data.txt desthost=localhost
> > > destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > provider=gsiftp policy=DEFAULT
> > >
> > >
> > > The desthost is localhost. That is correct. But why does the
> > > destdir have /gpfs in it? My localhost is VM. It does not have
> > > access to gpfs.
> > >
> > >
> > > The files for this run are in ~jonmon/run.0002 on the ci machines.
> > >
> > >
> > > On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:
> > >
> > > > I was going to suggest that same as Sarah pointed out,
> > > > suspecting
> > > > that the short-form of single_file_mapper is not correctly
> > > > parsing
> > > > the GSIFTP URI. If thats the case, please file it as a bug, Jon.
> > > >
> > > >
> > > > Can you try both forms of the mapper syntax with a known-good
> > > > URI?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > - Mike
> > > >
> > > >
> > > >
> > > > __________________________________________________________________
> > > >         Thanks. I just verified that something maybe wrong with
> > > >         the path. I could not globus-url-copy that URI so I will
> > > >         continue to investigate
> > > >
> > > >         On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> > > >
> > > >                 i've used this successfully in the past:
> > > >
> > > >                 file mybrain<single_file_mapper;
> > > >                 file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> > > >
> > > >                 it's been a while though...haven't tested with
> > > >                 the
> > > >                 latest swift...
> > > >
> > > >                 On Tue, Jul 12, 2011 at 1:59 PM, Jonathan
> > > >                 Monette <jonmon at utexas.edu> wrote:
> > > >                         How does this syntax look when mapping
> > > >                         with the GSIURI? How does it work? I am
> > > >                         still getting errors when trying to map
> > > >                         a
> > > >                         file with a GSIURI. It says file not
> > > >                         found. Maybe I have a different
> > > >                         understanding on how it works.
> > > >
> > > >                         The files and the logs are in
> > > >                         ~jonmon/run.0001 on the ci machines.
> > > >
> > > >
> > > >                         On Jul 12, 2011, at 2:16 AM, Mihael
> > > >                         Hategan wrote:
> > > >
> > > >                         > You have a colon after the host name
> > > >                         > but
> > > >                         no port. Either remove the
> > > >                         > colon or put a number after it.
> > > >                         >
> > > >                         > On Mon, 2011-07-11 at 22:45 -0500,
> > > >                         Jonathan Monette wrote:
> > > >                         >> I actually meant to send this to
> > > >                         swift-devel.
> > > >                         >>
> > > >                         >>
> > > >                         >> Here is the background to the
> > > >                         >> problem.
> > > >                          I have data on PADS, I am
> > > >                         >> executing Swift on a VM, and I want
> > > >                         >> to
> > > >                         use OSG to compute with the
> > > >                         >> data. Before Mike left for vacation
> > > >                         >> he
> > > >                         said that you can map data in
> > > >                         >> Swift using the GSIURI scheme but he
> > > >                         did not tell me how. What I did
> > > >                         >> below is
> > > >                         >>
> > > >                         >>
> > > >                         >> file
> > > >                         >>
> > > >                         data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> > > >                         >>
> > > >                         >>
> > > >                         >> This does not seem to work in release
> > > >                         0.92.1 as the error below
> > > >                         >> shows. How do you map data in Swift
> > > >                         using the GSIURI scheme?
> > > >                         >>
> > > >                         >> Begin forwarded message:
> > > >                         >>
> > > >                         >>> From: Jonathan Monette
> > > >                         <jonmon at utexas.edu>
> > > >                         >>>
> > > >                         >>> Date: July 11, 2011 2:30:34 PM CDT
> > > >                         >>>
> > > >                         >>> To: swift-devel Devel
> > > >                         <swift-devel at ci.uchicago.edu>
> > > >                         >>>
> > > >                         >>> Cc: Mihael Hategan Hategan
> > > >                         <hategan at mcs.anl.gov>
> > > >                         >>>
> > > >                         >>> Subject: NumberFormatException
> > > >                         >>>
> > > >                         >>>
> > > >                         >>> Mihael,
> > > >                         >>>  I am getting this error using
> > > >                         >>>  release
> > > >                         0.92.1.
> > > >                         >>>
> > > >                         >>> 2011-07-11 19:24:44,395+0000 INFO
> > > >                          unknown RUNID
> > > >                         >>> id=run:20110711-1924-c944yl9c
> > > >                         >>> 2011-07-11 19:24:44,508+0000 DEBUG
> > > >                         VDL2ExecutionContext vdl:new @
> > > >                         >>> script.kml, line: 69:
> > > >                         java.lang.RuntimeException:
> > > >                         >>> java.lang.NumberFormatException: For
> > > >                         input string: ""
> > > >                         >>> java.lang.RuntimeException:
> > > >                         java.lang.NumberFormatException: For
> > > >                         >>> input string: ""
> > > >                         >>> Caused by:
> > > >                         >>> java.lang.RuntimeException:
> > > >                         >>> java.lang.NumberFormatException: For
> > > >                         input string: ""
> > > >                         >>> at
> > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> > > >                         >>> at
> > > >                         org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> > > >                         >>> at
> > > >                         org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> > > >                         >>> at
> > > >                         org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > >                         >>> at
> > > >                         org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > > >                         >>> at
> > > >                         edu.emory.mathcs.backport.java.util.concurrent.Executors
> > > >                         >>>
> > > >                         $RunnableAdapter.call(Executors.java:431)
> > > >                         >>> at
> > > >                         >>>
> > > >                         edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > >                         >>> at
> > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > >                         >>>
> > > >                         $Worker.runTask(ThreadPoolExecutor.java:643)
> > > >                         >>> at
> > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > >                         >>>
> > > >                         $Worker.run(ThreadPoolExecutor.java:668)
> > > >                         >>> at
> > > >                         java.lang.Thread.run(Thread.java:636)
> > > >                         >>> Caused by:
> > > >                         java.lang.NumberFormatException: For
> > > >                         input
> > > >                         string: ""
> > > >                         >>> at
> > > >                         >>>
> > > >                         java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > > >                         >>> at
> > > >                         java.lang.Integer.parseInt(Integer.java:493)
> > > >                         >>> at
> > > >                         java.lang.Integer.parseInt(Integer.java:514)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> > > >                         >>> at
> > > >                         >>>
> > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> > > >                         >>> at
> > > >                         org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> > > >                         >>> at
> > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> > > >                         >>> ... 63 more
> > > >                         >>>
> > > >                         >>> The files needed for this run are
> > > >                         located in ~jonmon/run.0000 on the
> > > >                         >>> ci machines.
> > > >                         >>
> > > >                         >>
> > > >                         >
> > > >                         >
> > > >                         >
> > > >                         _______________________________________________
> > > >                         > Swift-devel mailing list
> > > >                         > Swift-devel at ci.uchicago.edu
> > > >                         > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >                         _______________________________________________
> > > >                         Swift-devel mailing list
> > > >                         Swift-devel at ci.uchicago.edu
> > > >                         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >
> > > >
> > > >
> > > >                 --
> > > >                 Sarah Kenny
> > > >                 Programmer
> > > >                 University of Chicago, Computation Institute
> > > >                 University of California Irvine, Dept. of
> > > >                 Neurology
> > > >                 773-818-8300
> > > >
> > > >                 _______________________________________________
> > > >                 Swift-devel mailing list
> > > >                 Swift-devel at ci.uchicago.edu
> > > >                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >
> > > >
> > > >         _______________________________________________
> > > >         Swift-devel mailing list
> > > >         Swift-devel at ci.uchicago.edu
> > > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >
> > > >
> > > > --
> > > > Michael Wilde
> > > > Computation Institute, University of Chicago
> > > > Mathematics and Computer Science Division
> > > > Argonne National Laboratory
> > > >
> > > >
> > >
> > >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Wed Jul 13 19:19:58 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 13 Jul 2011 19:19:58 -0500
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <1700525737.93869.1310602161459.JavaMail.root@zimbra.anl.gov>
References: <1700525737.93869.1310602161459.JavaMail.root@zimbra.anl.gov>
Message-ID: <1310602798.2681.1.camel@blabla>

Sadly there are many other things that will fail if the swift client is
unreachable from the outside world. Reachability (at least on some
number of ports - TCP_PORT_RANGE) is a prerequisite for swift.

On Wed, 2011-07-13 at 19:09 -0500, Michael Wilde wrote:
> If this is the same error I think it is, its likely due to the fact that Swift was trying to use 3rd party transfer with the Swift client on an EC2 VM which was unreachable from the PADS GridFTP server.
> 
> It raised some interesting issues on how Swift tried to do transfers from a file mapped to a gsiftp:// URI and a local execution site which I think are still unresolved but were the likely cause of this error.
> 
> I think Jon is reworking this test case to use only remote GridFTp-enabled sites outside of EC2 as the execution sites, and the correct DNS or IP name for inbound connections back to Swift.
> 
> - Mike
> 
> 
> ----- Original Message -----
> > The full log will help here, but the "Reply wait timeout" exception is
> > not necessarily an indication of a bad path (though I'm not saying
> > it's
> > not).
> > 
> > In the meantime, can you check that you have GLOBUS_HOSTNAME set
> > properly and that your client machine is not behind a firewall?
> > 
> > On Wed, 2011-07-13 at 13:00 -0500, Jonathan Monette wrote:
> > > Mike mentioned that this path is correct and makes sense. I was able
> > > to use the globus-url-copy from the swift bin directory and make a
> > > copy from that GSIURI. The error that I am receiving is
> > >
> > >
> > > Caused by: Exception in getFile
> > > Caused by:
> > > org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> > > Exception in getFile
> > > Caused by: org.globus.ftp.exception.ServerException: Reply wait
> > > timeout. (error code 4)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > at
> > > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > > at
> > > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > > at
> > > org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
> > > at
> > > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > at
> > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > > at edu.emory.mathcs.backport.java.util.concurrent.Executors
> > > $RunnableAdapter.call(Executors.java:431)
> > > at
> > > edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > $Worker.runTask(ThreadPoolExecutor.java:643)
> > > at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > $Worker.run(ThreadPoolExecutor.java:668)
> > > at java.lang.Thread.run(Thread.java:636)
> > >
> > >
> > > I don't know if I mentioned this but this is with the 0.92.1
> > > release.
> > >
> > >
> > > On Jul 13, 2011, at 11:29 AM, Jonathan Monette wrote:
> > >
> > > > I am starting to believe that this maybe a bug. I have this line
> > > > in
> > > > the log.
> > > >
> > > >
> > > > 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile
> > > > FILE_STAGE_IN_START file=data.txt
> > > > srchost=gridftp.pads.ci.uchicago.edu:2811
> > > > srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > > srcname=data.txt desthost=localhost
> > > > destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > > provider=gsiftp policy=DEFAULT
> > > >
> > > >
> > > > The desthost is localhost. That is correct. But why does the
> > > > destdir have /gpfs in it? My localhost is VM. It does not have
> > > > access to gpfs.
> > > >
> > > >
> > > > The files for this run are in ~jonmon/run.0002 on the ci machines.
> > > >
> > > >
> > > > On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:
> > > >
> > > > > I was going to suggest that same as Sarah pointed out,
> > > > > suspecting
> > > > > that the short-form of single_file_mapper is not correctly
> > > > > parsing
> > > > > the GSIFTP URI. If thats the case, please file it as a bug, Jon.
> > > > >
> > > > >
> > > > > Can you try both forms of the mapper syntax with a known-good
> > > > > URI?
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > >
> > > > > - Mike
> > > > >
> > > > >
> > > > >
> > > > > __________________________________________________________________
> > > > >         Thanks. I just verified that something maybe wrong with
> > > > >         the path. I could not globus-url-copy that URI so I will
> > > > >         continue to investigate
> > > > >
> > > > >         On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> > > > >
> > > > >                 i've used this successfully in the past:
> > > > >
> > > > >                 file mybrain<single_file_mapper;
> > > > >                 file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> > > > >
> > > > >                 it's been a while though...haven't tested with
> > > > >                 the
> > > > >                 latest swift...
> > > > >
> > > > >                 On Tue, Jul 12, 2011 at 1:59 PM, Jonathan
> > > > >                 Monette <jonmon at utexas.edu> wrote:
> > > > >                         How does this syntax look when mapping
> > > > >                         with the GSIURI? How does it work? I am
> > > > >                         still getting errors when trying to map
> > > > >                         a
> > > > >                         file with a GSIURI. It says file not
> > > > >                         found. Maybe I have a different
> > > > >                         understanding on how it works.
> > > > >
> > > > >                         The files and the logs are in
> > > > >                         ~jonmon/run.0001 on the ci machines.
> > > > >
> > > > >
> > > > >                         On Jul 12, 2011, at 2:16 AM, Mihael
> > > > >                         Hategan wrote:
> > > > >
> > > > >                         > You have a colon after the host name
> > > > >                         > but
> > > > >                         no port. Either remove the
> > > > >                         > colon or put a number after it.
> > > > >                         >
> > > > >                         > On Mon, 2011-07-11 at 22:45 -0500,
> > > > >                         Jonathan Monette wrote:
> > > > >                         >> I actually meant to send this to
> > > > >                         swift-devel.
> > > > >                         >>
> > > > >                         >>
> > > > >                         >> Here is the background to the
> > > > >                         >> problem.
> > > > >                          I have data on PADS, I am
> > > > >                         >> executing Swift on a VM, and I want
> > > > >                         >> to
> > > > >                         use OSG to compute with the
> > > > >                         >> data. Before Mike left for vacation
> > > > >                         >> he
> > > > >                         said that you can map data in
> > > > >                         >> Swift using the GSIURI scheme but he
> > > > >                         did not tell me how. What I did
> > > > >                         >> below is
> > > > >                         >>
> > > > >                         >>
> > > > >                         >> file
> > > > >                         >>
> > > > >                         data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> > > > >                         >>
> > > > >                         >>
> > > > >                         >> This does not seem to work in release
> > > > >                         0.92.1 as the error below
> > > > >                         >> shows. How do you map data in Swift
> > > > >                         using the GSIURI scheme?
> > > > >                         >>
> > > > >                         >> Begin forwarded message:
> > > > >                         >>
> > > > >                         >>> From: Jonathan Monette
> > > > >                         <jonmon at utexas.edu>
> > > > >                         >>>
> > > > >                         >>> Date: July 11, 2011 2:30:34 PM CDT
> > > > >                         >>>
> > > > >                         >>> To: swift-devel Devel
> > > > >                         <swift-devel at ci.uchicago.edu>
> > > > >                         >>>
> > > > >                         >>> Cc: Mihael Hategan Hategan
> > > > >                         <hategan at mcs.anl.gov>
> > > > >                         >>>
> > > > >                         >>> Subject: NumberFormatException
> > > > >                         >>>
> > > > >                         >>>
> > > > >                         >>> Mihael,
> > > > >                         >>>  I am getting this error using
> > > > >                         >>>  release
> > > > >                         0.92.1.
> > > > >                         >>>
> > > > >                         >>> 2011-07-11 19:24:44,395+0000 INFO
> > > > >                          unknown RUNID
> > > > >                         >>> id=run:20110711-1924-c944yl9c
> > > > >                         >>> 2011-07-11 19:24:44,508+0000 DEBUG
> > > > >                         VDL2ExecutionContext vdl:new @
> > > > >                         >>> script.kml, line: 69:
> > > > >                         java.lang.RuntimeException:
> > > > >                         >>> java.lang.NumberFormatException: For
> > > > >                         input string: ""
> > > > >                         >>> java.lang.RuntimeException:
> > > > >                         java.lang.NumberFormatException: For
> > > > >                         >>> input string: ""
> > > > >                         >>> Caused by:
> > > > >                         >>> java.lang.RuntimeException:
> > > > >                         >>> java.lang.NumberFormatException: For
> > > > >                         input string: ""
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > >                         >>> at
> > > > >                         org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > > > >                         >>> at
> > > > >                         edu.emory.mathcs.backport.java.util.concurrent.Executors
> > > > >                         >>>
> > > > >                         $RunnableAdapter.call(Executors.java:431)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > >                         >>> at
> > > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > >                         >>>
> > > > >                         $Worker.runTask(ThreadPoolExecutor.java:643)
> > > > >                         >>> at
> > > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > >                         >>>
> > > > >                         $Worker.run(ThreadPoolExecutor.java:668)
> > > > >                         >>> at
> > > > >                         java.lang.Thread.run(Thread.java:636)
> > > > >                         >>> Caused by:
> > > > >                         java.lang.NumberFormatException: For
> > > > >                         input
> > > > >                         string: ""
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > > > >                         >>> at
> > > > >                         java.lang.Integer.parseInt(Integer.java:493)
> > > > >                         >>> at
> > > > >                         java.lang.Integer.parseInt(Integer.java:514)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> > > > >                         >>> at
> > > > >                         >>>
> > > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> > > > >                         >>> at
> > > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> > > > >                         >>> ... 63 more
> > > > >                         >>>
> > > > >                         >>> The files needed for this run are
> > > > >                         located in ~jonmon/run.0000 on the
> > > > >                         >>> ci machines.
> > > > >                         >>
> > > > >                         >>
> > > > >                         >
> > > > >                         >
> > > > >                         >
> > > > >                         _______________________________________________
> > > > >                         > Swift-devel mailing list
> > > > >                         > Swift-devel at ci.uchicago.edu
> > > > >                         > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >                         _______________________________________________
> > > > >                         Swift-devel mailing list
> > > > >                         Swift-devel at ci.uchicago.edu
> > > > >                         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >                 --
> > > > >                 Sarah Kenny
> > > > >                 Programmer
> > > > >                 University of Chicago, Computation Institute
> > > > >                 University of California Irvine, Dept. of
> > > > >                 Neurology
> > > > >                 773-818-8300
> > > > >
> > > > >                 _______________________________________________
> > > > >                 Swift-devel mailing list
> > > > >                 Swift-devel at ci.uchicago.edu
> > > > >                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >
> > > > >
> > > > >         _______________________________________________
> > > > >         Swift-devel mailing list
> > > > >         Swift-devel at ci.uchicago.edu
> > > > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Michael Wilde
> > > > > Computation Institute, University of Chicago
> > > > > Mathematics and Computer Science Division
> > > > > Argonne National Laboratory
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 




From wilde at mcs.anl.gov  Wed Jul 13 22:28:57 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 13 Jul 2011 22:28:57 -0500 (CDT)
Subject: [Swift-devel] Fwd: NumberFormatException
In-Reply-To: <1310602798.2681.1.camel@blabla>
Message-ID: <1379219459.94067.1310614137494.JavaMail.root@zimbra.anl.gov>

Right; in this case, the client was physically reachable by an IP that was not listed in ifconfig, and GLOBUS_HOSTNAME was not specified.  We still have some fiddling to do before its clear if Swift works in this particular configuration and transfer mode.

- Mike 

----- Original Message -----
> Sadly there are many other things that will fail if the swift client
> is
> unreachable from the outside world. Reachability (at least on some
> number of ports - TCP_PORT_RANGE) is a prerequisite for swift.
> 
> On Wed, 2011-07-13 at 19:09 -0500, Michael Wilde wrote:
> > If this is the same error I think it is, its likely due to the fact
> > that Swift was trying to use 3rd party transfer with the Swift
> > client on an EC2 VM which was unreachable from the PADS GridFTP
> > server.
> >
> > It raised some interesting issues on how Swift tried to do transfers
> > from a file mapped to a gsiftp:// URI and a local execution site
> > which I think are still unresolved but were the likely cause of this
> > error.
> >
> > I think Jon is reworking this test case to use only remote
> > GridFTp-enabled sites outside of EC2 as the execution sites, and the
> > correct DNS or IP name for inbound connections back to Swift.
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> > > The full log will help here, but the "Reply wait timeout"
> > > exception is
> > > not necessarily an indication of a bad path (though I'm not saying
> > > it's
> > > not).
> > >
> > > In the meantime, can you check that you have GLOBUS_HOSTNAME set
> > > properly and that your client machine is not behind a firewall?
> > >
> > > On Wed, 2011-07-13 at 13:00 -0500, Jonathan Monette wrote:
> > > > Mike mentioned that this path is correct and makes sense. I was
> > > > able
> > > > to use the globus-url-copy from the swift bin directory and make
> > > > a
> > > > copy from that GSIURI. The error that I am receiving is
> > > >
> > > >
> > > > Caused by: Exception in getFile
> > > > Caused by:
> > > > org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> > > > Exception in getFile
> > > > Caused by: org.globus.ftp.exception.ServerException: Reply wait
> > > > timeout. (error code 4)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.functions.KException.function(KException.java:29)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > at
> > > > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > > > at
> > > > org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > > > at
> > > > org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
> > > > at
> > > > org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > at
> > > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > > > at edu.emory.mathcs.backport.java.util.concurrent.Executors
> > > > $RunnableAdapter.call(Executors.java:431)
> > > > at
> > > > edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > at
> > > > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > $Worker.runTask(ThreadPoolExecutor.java:643)
> > > > at
> > > > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > $Worker.run(ThreadPoolExecutor.java:668)
> > > > at java.lang.Thread.run(Thread.java:636)
> > > >
> > > >
> > > > I don't know if I mentioned this but this is with the 0.92.1
> > > > release.
> > > >
> > > >
> > > > On Jul 13, 2011, at 11:29 AM, Jonathan Monette wrote:
> > > >
> > > > > I am starting to believe that this maybe a bug. I have this
> > > > > line
> > > > > in
> > > > > the log.
> > > > >
> > > > >
> > > > > 2011-07-13 16:17:50,835+0000 DEBUG vdl:dostageinfile
> > > > > FILE_STAGE_IN_START file=data.txt
> > > > > srchost=gridftp.pads.ci.uchicago.edu:2811
> > > > > srcdir=/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > > > srcname=data.txt desthost=localhost
> > > > > destdir=script-20110713-1617-lzk6sxj3/shared/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test
> > > > > provider=gsiftp policy=DEFAULT
> > > > >
> > > > >
> > > > > The desthost is localhost. That is correct. But why does the
> > > > > destdir have /gpfs in it? My localhost is VM. It does not have
> > > > > access to gpfs.
> > > > >
> > > > >
> > > > > The files for this run are in ~jonmon/run.0002 on the ci
> > > > > machines.
> > > > >
> > > > >
> > > > > On Jul 13, 2011, at 9:13 AM, Michael Wilde wrote:
> > > > >
> > > > > > I was going to suggest that same as Sarah pointed out,
> > > > > > suspecting
> > > > > > that the short-form of single_file_mapper is not correctly
> > > > > > parsing
> > > > > > the GSIFTP URI. If thats the case, please file it as a bug,
> > > > > > Jon.
> > > > > >
> > > > > >
> > > > > > Can you try both forms of the mapper syntax with a
> > > > > > known-good
> > > > > > URI?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > - Mike
> > > > > >
> > > > > >
> > > > > >
> > > > > > __________________________________________________________________
> > > > > >         Thanks. I just verified that something maybe wrong
> > > > > >         with
> > > > > >         the path. I could not globus-url-copy that URI so I
> > > > > >         will
> > > > > >         continue to investigate
> > > > > >
> > > > > >         On Jul 12, 2011, at 4:59 PM, Sarah Kenny wrote:
> > > > > >
> > > > > >                 i've used this successfully in the past:
> > > > > >
> > > > > >                 file mybrain<single_file_mapper;
> > > > > >                 file="gsiftp://calero.bsd.uchicago.edu/gpfs/pads/projects/stroke_recovery/dude.mgz")>;
> > > > > >
> > > > > >                 it's been a while though...haven't tested
> > > > > >                 with
> > > > > >                 the
> > > > > >                 latest swift...
> > > > > >
> > > > > >                 On Tue, Jul 12, 2011 at 1:59 PM, Jonathan
> > > > > >                 Monette <jonmon at utexas.edu> wrote:
> > > > > >                         How does this syntax look when
> > > > > >                         mapping
> > > > > >                         with the GSIURI? How does it work? I
> > > > > >                         am
> > > > > >                         still getting errors when trying to
> > > > > >                         map
> > > > > >                         a
> > > > > >                         file with a GSIURI. It says file not
> > > > > >                         found. Maybe I have a different
> > > > > >                         understanding on how it works.
> > > > > >
> > > > > >                         The files and the logs are in
> > > > > >                         ~jonmon/run.0001 on the ci machines.
> > > > > >
> > > > > >
> > > > > >                         On Jul 12, 2011, at 2:16 AM, Mihael
> > > > > >                         Hategan wrote:
> > > > > >
> > > > > >                         > You have a colon after the host
> > > > > >                         > name
> > > > > >                         > but
> > > > > >                         no port. Either remove the
> > > > > >                         > colon or put a number after it.
> > > > > >                         >
> > > > > >                         > On Mon, 2011-07-11 at 22:45 -0500,
> > > > > >                         Jonathan Monette wrote:
> > > > > >                         >> I actually meant to send this to
> > > > > >                         swift-devel.
> > > > > >                         >>
> > > > > >                         >>
> > > > > >                         >> Here is the background to the
> > > > > >                         >> problem.
> > > > > >                          I have data on PADS, I am
> > > > > >                         >> executing Swift on a VM, and I
> > > > > >                         >> want
> > > > > >                         >> to
> > > > > >                         use OSG to compute with the
> > > > > >                         >> data. Before Mike left for
> > > > > >                         >> vacation
> > > > > >                         >> he
> > > > > >                         said that you can map data in
> > > > > >                         >> Swift using the GSIURI scheme but
> > > > > >                         >> he
> > > > > >                         did not tell me how. What I did
> > > > > >                         >> below is
> > > > > >                         >>
> > > > > >                         >>
> > > > > >                         >> file
> > > > > >                         >>
> > > > > >                         data<"gsiftp://stor01.pads.ci.uchicago.edu:/gpfs/pads/projects/CI-CCR000013/jonmon/Swift/tests/cat_test/data.txt">;
> > > > > >                         >>
> > > > > >                         >>
> > > > > >                         >> This does not seem to work in
> > > > > >                         >> release
> > > > > >                         0.92.1 as the error below
> > > > > >                         >> shows. How do you map data in
> > > > > >                         >> Swift
> > > > > >                         using the GSIURI scheme?
> > > > > >                         >>
> > > > > >                         >> Begin forwarded message:
> > > > > >                         >>
> > > > > >                         >>> From: Jonathan Monette
> > > > > >                         <jonmon at utexas.edu>
> > > > > >                         >>>
> > > > > >                         >>> Date: July 11, 2011 2:30:34 PM
> > > > > >                         >>> CDT
> > > > > >                         >>>
> > > > > >                         >>> To: swift-devel Devel
> > > > > >                         <swift-devel at ci.uchicago.edu>
> > > > > >                         >>>
> > > > > >                         >>> Cc: Mihael Hategan Hategan
> > > > > >                         <hategan at mcs.anl.gov>
> > > > > >                         >>>
> > > > > >                         >>> Subject: NumberFormatException
> > > > > >                         >>>
> > > > > >                         >>>
> > > > > >                         >>> Mihael,
> > > > > >                         >>>  I am getting this error using
> > > > > >                         >>>  release
> > > > > >                         0.92.1.
> > > > > >                         >>>
> > > > > >                         >>> 2011-07-11 19:24:44,395+0000
> > > > > >                         >>> INFO
> > > > > >                          unknown RUNID
> > > > > >                         >>> id=run:20110711-1924-c944yl9c
> > > > > >                         >>> 2011-07-11 19:24:44,508+0000
> > > > > >                         >>> DEBUG
> > > > > >                         VDL2ExecutionContext vdl:new @
> > > > > >                         >>> script.kml, line: 69:
> > > > > >                         java.lang.RuntimeException:
> > > > > >                         >>> java.lang.NumberFormatException:
> > > > > >                         >>> For
> > > > > >                         input string: ""
> > > > > >                         >>> java.lang.RuntimeException:
> > > > > >                         java.lang.NumberFormatException: For
> > > > > >                         >>> input string: ""
> > > > > >                         >>> Caused by:
> > > > > >                         >>> java.lang.RuntimeException:
> > > > > >                         >>> java.lang.NumberFormatException:
> > > > > >                         >>> For
> > > > > >                         input string: ""
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:109)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.griphyn.vdl.mapping.file.SingleFileMapper.existing(SingleFileMapper.java:24)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:97)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:75)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.griphyn.vdl.mapping.RootDataNode.innerInit(RootDataNode.java:61)
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.mapping.RootDataNode.init(RootDataNode.java:37)
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.karajan.lib.New.function(New.java:126)
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.childCompleted(UserDefinedElement.java:290)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.childCompleted(SequentialImplicitExecutionUDE.java:85)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:45)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.functions.Map_Map.post(Map_Map.java:55)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > > >                         >>> at
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Each.post(Each.java:31)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:192)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
> > > > > >                         >>> at
> > > > > >                         edu.emory.mathcs.backport.java.util.concurrent.Executors
> > > > > >                         >>>
> > > > > >                         $RunnableAdapter.call(Executors.java:431)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > > >                         >>> at
> > > > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > > >                         >>>
> > > > > >                         $Worker.runTask(ThreadPoolExecutor.java:643)
> > > > > >                         >>> at
> > > > > >                         edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> > > > > >                         >>>
> > > > > >                         $Worker.run(ThreadPoolExecutor.java:668)
> > > > > >                         >>> at
> > > > > >                         java.lang.Thread.run(Thread.java:636)
> > > > > >                         >>> Caused by:
> > > > > >                         java.lang.NumberFormatException: For
> > > > > >                         input
> > > > > >                         string: ""
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> > > > > >                         >>> at
> > > > > >                         java.lang.Integer.parseInt(Integer.java:493)
> > > > > >                         >>> at
> > > > > >                         java.lang.Integer.parseInt(Integer.java:514)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.parse(ServiceContactImpl.java:90)
> > > > > >                         >>> at
> > > > > >                         >>>
> > > > > >                         org.globus.cog.abstraction.impl.common.task.ServiceContactImpl.<init>(ServiceContactImpl.java:27)
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.mapping.AbsFile.getFileResource(AbsFile.java:84)
> > > > > >                         >>> at
> > > > > >                         org.griphyn.vdl.mapping.AbsFile.exists(AbsFile.java:99)
> > > > > >                         >>> ... 63 more
> > > > > >                         >>>
> > > > > >                         >>> The files needed for this run
> > > > > >                         >>> are
> > > > > >                         located in ~jonmon/run.0000 on the
> > > > > >                         >>> ci machines.
> > > > > >                         >>
> > > > > >                         >>
> > > > > >                         >
> > > > > >                         >
> > > > > >                         >
> > > > > >                         _______________________________________________
> > > > > >                         > Swift-devel mailing list
> > > > > >                         > Swift-devel at ci.uchicago.edu
> > > > > >                         > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >                         _______________________________________________
> > > > > >                         Swift-devel mailing list
> > > > > >                         Swift-devel at ci.uchicago.edu
> > > > > >                         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >                 --
> > > > > >                 Sarah Kenny
> > > > > >                 Programmer
> > > > > >                 University of Chicago, Computation Institute
> > > > > >                 University of California Irvine, Dept. of
> > > > > >                 Neurology
> > > > > >                 773-818-8300
> > > > > >
> > > > > >                 _______________________________________________
> > > > > >                 Swift-devel mailing list
> > > > > >                 Swift-devel at ci.uchicago.edu
> > > > > >                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >
> > > > > >
> > > > > >         _______________________________________________
> > > > > >         Swift-devel mailing list
> > > > > >         Swift-devel at ci.uchicago.edu
> > > > > >         https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Michael Wilde
> > > > > > Computation Institute, University of Chicago
> > > > > > Mathematics and Computer Science Division
> > > > > > Argonne National Laboratory
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jonmon at utexas.edu  Thu Jul 14 13:14:48 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Thu, 14 Jul 2011 13:14:48 -0500
Subject: [Swift-devel] OSG hang
Message-ID: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>

Hello,
   I am trying to submit jobs from an Amazon VM to an OSG site.  I have tried Engage VO sites with a proxy created by both grid-proxy-init and voms-proxy-init.  When running swift it says it submitted the job but it looks like nothing is executing.  This happens when using fork, condor, and pbs. I can successfully execute the script from communicado to OSG using fork with barely any wait time.  

The files are located in ~jonmon/OSG.0000 on the ci machines.  

I have filed a bug in bugzilla.
Bug 471 - OSG hangs from VM

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110714/a6026b32/attachment.html>

From wilde at mcs.anl.gov  Thu Jul 14 13:22:56 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 14 Jul 2011 13:22:56 -0500 (CDT)
Subject: [Swift-devel] OSG hang
In-Reply-To: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>
Message-ID: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>

Try this: 


export GLOBUS_HOSTNAME=externalIPAddr # set it to the 50.x.y.z addr 
export GLOBUS_TCP_PORT_RANGE=50000,51000 
export GLOBUS_TCP_SOURCE_RANGE=50000,51000 


Then test both globus-url-copy and globusrun from swift/bin (you will need simple RSL for globusrun) 


Then test Swift manually from the VM 


Then test with a Karajan script 


Then test the Swift script 


Use a single sleep job, eg sleep 1234, so you can spot it running on the head node. 


Mihael, can you help walk Jon through the above steps or suggest another approach? 


Thanks, 


Mike 



----- Original Message -----


Hello, 
I am trying to submit jobs from an Amazon VM to an OSG site. I have tried Engage VO sites with a proxy created by both grid-proxy-init and voms-proxy-init. When running swift it says it submitted the job but it looks like nothing is executing. This happens when using fork, condor, and pbs. I can successfully execute the script from communicado to OSG using fork with barely any wait time. 


The files are located in ~jonmon/OSG.0000 on the ci machines. 


I have filed a bug in bugzilla. 
Bug 471 - OSG hangs from VM 




-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110714/32db5369/attachment.html>

From hategan at mcs.anl.gov  Thu Jul 14 13:24:29 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 11:24:29 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>
References: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>
Message-ID: <1310667869.6072.6.camel@blabla>

On Thu, 2011-07-14 at 13:14 -0500, Jonathan Monette wrote:
> Hello,
>    I am trying to submit jobs from an Amazon VM to an OSG site.  I
> have tried Engage VO sites with a proxy created by both
> grid-proxy-init and voms-proxy-init.  When running swift it says it
> submitted the job but it looks like nothing is executing.

Is nothing really executing or is it swift not saying anything was done?

The distinction is important because the latter may mean that the
service isn't telling swift that stuff was done (this is what I was
alluding to yesterday when I said that if GLOBUS_HOSTNAME isn't set
properly, lots of things will break).

>   This happens when using fork, condor, and pbs. I can successfully
> execute the script from communicado to OSG using fork with barely any
> wait time.  
> 
> 
> The files are located in ~jonmon/OSG.0000 on the ci machines.  
> 
> 
> I have filed a bug in bugzilla.
> Bug 471 - OSG hangs from VM
> 
> 




From hategan at mcs.anl.gov  Thu Jul 14 13:25:46 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 11:25:46 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
Message-ID: <1310667946.6072.7.camel@blabla>

On Thu, 2011-07-14 at 13:22 -0500, Michael Wilde wrote:
> Try this:
> 
> 
> export GLOBUS_HOSTNAME=externalIPAddr # set it to the 50.x.y.z addr
> export GLOBUS_TCP_PORT_RANGE=50000,51000
> export GLOBUS_TCP_SOURCE_RANGE=50000,51000
> 
> 
> Then test both globus-url-copy and globusrun from swift/bin (you will
> need simple RSL for globusrun)
> 
> 
> Then test Swift manually from the VM
> 
> 
> Then test with a Karajan script
> 
> 
> Then test the Swift script
> 
> 
> Use a single sleep job, eg sleep 1234, so you can spot it running on
> the head node.
> 
> 
> Mihael, can you help walk Jon through the above steps or suggest
> another approach?

I will. The steps you mention are what I would try.





From jonmon at utexas.edu  Thu Jul 14 13:27:33 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Thu, 14 Jul 2011 13:27:33 -0500
Subject: [Swift-devel] OSG hang
In-Reply-To: <1310667869.6072.6.camel@blabla>
References: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>
	<1310667869.6072.6.camel@blabla>
Message-ID: <D3FFBEA1-DFFC-4BEA-9143-DB3CFAF539ED@utexas.edu>

How do I check the distinction?  The output file is not created, the log stops after the GridExec INFO message, and the Swift stdout just says "Progress:  Submitted:1" several times.

On Jul 14, 2011, at 1:24 PM, Mihael Hategan wrote:

> On Thu, 2011-07-14 at 13:14 -0500, Jonathan Monette wrote:
>> Hello,
>>   I am trying to submit jobs from an Amazon VM to an OSG site.  I
>> have tried Engage VO sites with a proxy created by both
>> grid-proxy-init and voms-proxy-init.  When running swift it says it
>> submitted the job but it looks like nothing is executing.
> 
> Is nothing really executing or is it swift not saying anything was done?
> 
> The distinction is important because the latter may mean that the
> service isn't telling swift that stuff was done (this is what I was
> alluding to yesterday when I said that if GLOBUS_HOSTNAME isn't set
> properly, lots of things will break).
> 
>>  This happens when using fork, condor, and pbs. I can successfully
>> execute the script from communicado to OSG using fork with barely any
>> wait time.  
>> 
>> 
>> The files are located in ~jonmon/OSG.0000 on the ci machines.  
>> 
>> 
>> I have filed a bug in bugzilla.
>> Bug 471 - OSG hangs from VM
>> 
>> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From jonmon at utexas.edu  Thu Jul 14 13:28:30 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Thu, 14 Jul 2011 13:28:30 -0500
Subject: [Swift-devel] OSG hang
In-Reply-To: <1310667946.6072.7.camel@blabla>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
	<1310667946.6072.7.camel@blabla>
Message-ID: <4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>

What is the distinction between testing swift manually from the VM and then test the Swift script?

On Jul 14, 2011, at 1:25 PM, Mihael Hategan wrote:

> On Thu, 2011-07-14 at 13:22 -0500, Michael Wilde wrote:
>> Try this:
>> 
>> 
>> export GLOBUS_HOSTNAME=externalIPAddr # set it to the 50.x.y.z addr
>> export GLOBUS_TCP_PORT_RANGE=50000,51000
>> export GLOBUS_TCP_SOURCE_RANGE=50000,51000
>> 
>> 
>> Then test both globus-url-copy and globusrun from swift/bin (you will
>> need simple RSL for globusrun)
>> 
>> 
>> Then test Swift manually from the VM
>> 
>> 
>> Then test with a Karajan script
>> 
>> 
>> Then test the Swift script
>> 
>> 
>> Use a single sleep job, eg sleep 1234, so you can spot it running on
>> the head node.
>> 
>> 
>> Mihael, can you help walk Jon through the above steps or suggest
>> another approach?
> 
> I will. The steps you mention are what I would try.
> 
> 
> 



From hategan at mcs.anl.gov  Thu Jul 14 13:30:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 11:30:34 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <D3FFBEA1-DFFC-4BEA-9143-DB3CFAF539ED@utexas.edu>
References: <628D1535-1C4F-481B-9A76-3709E8FE9C5C@utexas.edu>
	<1310667869.6072.6.camel@blabla>
	<D3FFBEA1-DFFC-4BEA-9143-DB3CFAF539ED@utexas.edu>
Message-ID: <1310668234.6072.8.camel@blabla>

On Thu, 2011-07-14 at 13:27 -0500, Jonathan Monette wrote:
> How do I check the distinction?  The output file is not created, the
> log stops after the GridExec INFO message, and the Swift stdout just
> says "Progress:  Submitted:1" several times.

Do what Mike says. Submit a long running job (sleep 1234 he suggested)
and look at the head node to see if it's running.




From hategan at mcs.anl.gov  Thu Jul 14 13:32:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 11:32:34 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
	<1310667946.6072.7.camel@blabla>
	<4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>
Message-ID: <1310668354.6072.10.camel@blabla>

On Thu, 2011-07-14 at 13:28 -0500, Jonathan Monette wrote:
> What is the distinction between testing swift manually from the VM and then test the Swift script?

I think Mike means that you should do a simple sleep script that you
would run by logging into the vm and typing "swift sleep.swift" on the
command line vs. whatever automated system you may have to launch your
real script.

More important, however, are the other steps.

> 
> On Jul 14, 2011, at 1:25 PM, Mihael Hategan wrote:
> 
> > On Thu, 2011-07-14 at 13:22 -0500, Michael Wilde wrote:
> >> Try this:
> >> 
> >> 
> >> export GLOBUS_HOSTNAME=externalIPAddr # set it to the 50.x.y.z addr
> >> export GLOBUS_TCP_PORT_RANGE=50000,51000
> >> export GLOBUS_TCP_SOURCE_RANGE=50000,51000
> >> 
> >> 
> >> Then test both globus-url-copy and globusrun from swift/bin (you will
> >> need simple RSL for globusrun)
> >> 
> >> 
> >> Then test Swift manually from the VM
> >> 
> >> 
> >> Then test with a Karajan script
> >> 
> >> 
> >> Then test the Swift script
> >> 
> >> 
> >> Use a single sleep job, eg sleep 1234, so you can spot it running on
> >> the head node.
> >> 
> >> 
> >> Mihael, can you help walk Jon through the above steps or suggest
> >> another approach?
> > 
> > I will. The steps you mention are what I would try.
> > 
> > 
> > 
> 




From wilde at mcs.anl.gov  Thu Jul 14 13:34:08 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 14 Jul 2011 13:34:08 -0500 (CDT)
Subject: [Swift-devel] OSG hang
In-Reply-To: <D3FFBEA1-DFFC-4BEA-9143-DB3CFAF539ED@utexas.edu>
Message-ID: <305252580.96042.1310668448329.JavaMail.root@zimbra.anl.gov>

make sleep 1234 the app call, and then probe the head node with a ps shell command via globus-job-run to see if the sleep is running.  Should run under the group account that the Engage VO is mapped to (run /usr/bin/id to find this)

In some cases, Globus can start the job but not repond back with completion status (ie if callback ports are not accessible) - hence the two _TCP_ env vars (see globus docs for this).

Im dropping out now and leaving you in Mihael's guidance for this.

- Mike

----- Original Message -----
> How do I check the distinction? The output file is not created, the
> log stops after the GridExec INFO message, and the Swift stdout just
> says "Progress: Submitted:1" several times.
> 
> On Jul 14, 2011, at 1:24 PM, Mihael Hategan wrote:
> 
> > On Thu, 2011-07-14 at 13:14 -0500, Jonathan Monette wrote:
> >> Hello,
> >>   I am trying to submit jobs from an Amazon VM to an OSG site. I
> >> have tried Engage VO sites with a proxy created by both
> >> grid-proxy-init and voms-proxy-init. When running swift it says it
> >> submitted the job but it looks like nothing is executing.
> >
> > Is nothing really executing or is it swift not saying anything was
> > done?
> >
> > The distinction is important because the latter may mean that the
> > service isn't telling swift that stuff was done (this is what I was
> > alluding to yesterday when I said that if GLOBUS_HOSTNAME isn't set
> > properly, lots of things will break).
> >
> >>  This happens when using fork, condor, and pbs. I can successfully
> >> execute the script from communicado to OSG using fork with barely
> >> any
> >> wait time.
> >>
> >>
> >> The files are located in ~jonmon/OSG.0000 on the ci machines.
> >>
> >>
> >> I have filed a bug in bugzilla.
> >> Bug 471 - OSG hangs from VM
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jonmon at utexas.edu  Thu Jul 14 14:02:40 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Thu, 14 Jul 2011 14:02:40 -0500
Subject: [Swift-devel] OSG hang
In-Reply-To: <1310668354.6072.10.camel@blabla>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
	<1310667946.6072.7.camel@blabla>
	<4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>
	<1310668354.6072.10.camel@blabla>
Message-ID: <ADBED7C4-2D45-4D59-B3EF-4575B4EE1CAF@utexas.edu>

Ok.  I ran globus-job-run tuscany.med.harvard.edu /bin/sh -c '/bin/ps -g engage'  from communicado before the swift script and got this:
PID TTY          TIME CMD
  341 ?        00:00:00 globus-job-mana
  447 ?        00:00:00 ps
13407 ?        00:00:00 globus-job-mana
14200 ?        00:00:00 perl
14202 ?        00:00:03 perl
17440 ?        00:00:00 condor_shadow

I then ran the swift script from the VM and ran the globus-job-run command from above and got:
  PID TTY          TIME CMD
 9145 ?        00:00:00 globus-gridftp-
 9493 ?        00:00:00 globus-job-mana
 9588 ?        00:00:00 bash
 9602 ?        00:00:00 sleep
11734 ?        00:00:00 globus-job-mana
11828 ?        00:00:00 ps
13407 ?        00:00:00 globus-job-mana
14200 ?        00:00:00 perl
14202 ?        00:00:03 perl
17440 ?        00:00:00 condor_shadow


So it does seem like the job get run on the head node.  When the job is finished on the head node(I checked several times with globus-job-run) Swift continues to execute(well it just prints the Submitted: 1 line over and over again).

So the next step is to set those environment variables Mike mentioned and try again?  

On Jul 14, 2011, at 1:32 PM, Mihael Hategan wrote:

> On Thu, 2011-07-14 at 13:28 -0500, Jonathan Monette wrote:
>> What is the distinction between testing swift manually from the VM and then test the Swift script?
> 
> I think Mike means that you should do a simple sleep script that you
> would run by logging into the vm and typing "swift sleep.swift" on the
> command line vs. whatever automated system you may have to launch your
> real script.
> 
> More important, however, are the other steps.
> 
>> 
>> On Jul 14, 2011, at 1:25 PM, Mihael Hategan wrote:
>> 
>>> On Thu, 2011-07-14 at 13:22 -0500, Michael Wilde wrote:
>>>> Try this:
>>>> 
>>>> 
>>>> export GLOBUS_HOSTNAME=externalIPAddr # set it to the 50.x.y.z addr
>>>> export GLOBUS_TCP_PORT_RANGE=50000,51000
>>>> export GLOBUS_TCP_SOURCE_RANGE=50000,51000
>>>> 
>>>> 
>>>> Then test both globus-url-copy and globusrun from swift/bin (you will
>>>> need simple RSL for globusrun)
>>>> 
>>>> 
>>>> Then test Swift manually from the VM
>>>> 
>>>> 
>>>> Then test with a Karajan script
>>>> 
>>>> 
>>>> Then test the Swift script
>>>> 
>>>> 
>>>> Use a single sleep job, eg sleep 1234, so you can spot it running on
>>>> the head node.
>>>> 
>>>> 
>>>> Mihael, can you help walk Jon through the above steps or suggest
>>>> another approach?
>>> 
>>> I will. The steps you mention are what I would try.
>>> 
>>> 
>>> 
>> 
> 
> 



From hategan at mcs.anl.gov  Thu Jul 14 14:07:06 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 12:07:06 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <ADBED7C4-2D45-4D59-B3EF-4575B4EE1CAF@utexas.edu>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
	<1310667946.6072.7.camel@blabla>
	<4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>
	<1310668354.6072.10.camel@blabla>
	<ADBED7C4-2D45-4D59-B3EF-4575B4EE1CAF@utexas.edu>
Message-ID: <1310670426.9855.2.camel@blabla>

On Thu, 2011-07-14 at 14:02 -0500, Jonathan Monette wrote:

> 
> So it does seem like the job get run on the head node.  When the job
> is finished on the head node(I checked several times with
> globus-job-run) Swift continues to execute(well it just prints the
> Submitted: 1 line over and over again).
> 
> So the next step is to set those environment variables Mike mentioned and try again?  

Yes. Swift won't work properly if those are not set properly. On most
system it guesses them, but not on weird systems.

Setting GLOBUS_HOSTNAME is probably enough though.




From ketancmaheshwari at gmail.com  Thu Jul 14 14:08:32 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 14 Jul 2011 14:08:32 -0500
Subject: [Swift-devel] Fwd: trunk FileNotFoundException
In-Reply-To: <1310521418.23008.5.camel@blabla>
References: <CAMUuviqEWZsCcdn5wqsxMx6pUJbC5gD0P8dSMBNgPFhwpm7S0A@mail.gmail.com>
	<CAMUuvipkRHxa3h4M1pko=y7+PUjz9JDV3GhZNTh7XEQcwykEJg@mail.gmail.com>
	<1310519744.23008.0.camel@blabla>
	<CAMUuvioMHZYAqKvMRqv=t=tMchWhgmFUzMiPGxuMY-L0HXkr7A@mail.gmail.com>
	<CAGMfakm++mdpLUyt9=2_R=aU8JsoFbMuKwB6Y0sZNejLs+qLjg@mail.gmail.com>
	<1310521418.23008.5.camel@blabla>
Message-ID: <CAMUuvip4M6tmjE2fk1vR7AM701=Vv6qroXvSZP-v5o8KL1yJEA@mail.gmail.com>

Hello,

It seems that the Swift on my trunk was a stale version. I updated and
rebuilt after which the NPE has gone.

Regards,
Ketan


On Tue, Jul 12, 2011 at 8:43 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> The NPE seems legit, and in quite the right spot considering the recent
> commits. But I was trying to figure out the thing with the workdir
> creation.
>
> In any event, does any of you have a simple test case for the NPE?
>
> On Tue, 2011-07-12 at 21:29 -0400, Glen Hocky wrote:
> > By the way, I also see this error, w/ pbs+coasters and on osg w/
> > condor, no provider staging
> >
> > On Tue, Jul 12, 2011 at 9:26 PM, Ketan Maheshwari
> > <ketancmaheshwari at gmail.com> wrote:
> >         No, provider staging is not on.
> >
> >         I am using local execution provider:
> >
> >         <pool handle="localhost">
> >             <filesystem provider="local" />
> >             <execution provider="local" />
> >             <workdirectory>/var/tmp/swift.workdir</workdirectory>
> >             <profile namespace="karajan"
> >         key="jobThrottle">0.20</profile>
> >         </pool>
> >
> >
> >
> >
> >
> >         On Tue, Jul 12, 2011 at 8:15 PM, Mihael Hategan
> >         <hategan at mcs.anl.gov> wrote:
> >                 Is provider staging on?
> >
> >
> >                 On Tue, 2011-07-12 at 14:16 -0500, Ketan Maheshwari
> >                 wrote:
> >                 > Mihael,
> >                 >
> >                 > I tried to further investigate the issue and from
> >                 the logs it seems
> >                 > that Swift is trying to execute the mkoffset app
> >                 before creating a
> >                 > jobs/ directory in workdir. Could it be that this is
> >                 an ordering
> >                 > issue. For instance, I see the following line:
> >                 >
> >                 > 2011-07-12 13:50:49,559-0500 DEBUG vdl:execute2
> >                 JOB_START
> >                 > jobid=mkoffset-mex8fvck tr=mkoffset
> >                 arguments=[200.0, 60.0]
> >                 >
> >
> tmpdir=postproc-20110712-1343-eczky6ob/jobs/m/mkoffset-mex8fvck
> >                 > host=localhost
> >                 >
> >                 > but do not see a createdir corresponding to above.
> >                 >
> >                 > I have ran this workflow successfully with 0.92.1
> >                 so, I am pretty sure
> >                 > that it works correctly as far as order of execution
> >                 is concerned.
> >                 >
> >                 > Thanks for any more insights into this.
> >                 >
> >                 > Regards,
> >                 > Ketan
> >                 >
> >                 > ---------- Forwarded message ----------
> >                 > From: Ketan Maheshwari <ketancmaheshwari at gmail.com>
> >                 > Date: Mon, Jul 11, 2011 at 10:31 PM
> >                 > Subject: trunk FileNotFoundException
> >                 > To: swift-user at ci.uchicago.edu
> >                 >
> >                 >
> >                 > Hello,
> >                 >
> >                 > Using Swift trunk, I am running the SCEC workflow
> >                 from Communicado
> >                 > using ranger, localhost and OSG resources.
> >                 >
> >                 > One particular app 'mkoffset' which is destined to
> >                 run on localhost is
> >                 > faulting with FileNotFoundException.
> >                 >
> >                 > The log does give information on its mapping and
> >                 when it gets
> >                 > 'cleared'.
> >                 >
> >                 > The config, tc, sites and log files for the run
> >                 could be found here:
> >                 > http://www.mcs.anl.gov/~ketan/files/bundle.tgz (log
> >                 is 90M, upload
> >                 > size exceeded!)
> >                 >
> >                 > The error stack that I am getting on stdout is:
> >                 >
> >                 > Progress:  time: Mon, 11 Jul 2011 22:16:38 -0500
> >                  Selecting site:390
> >                 > Stage in:16  Active:9  Checking status:1  Finished
> >                 successfully:36
> >                 > Failed but can retry:3
> >                 >
> >
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> >                 >
> >                 found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/stderr.txt
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
> >                 >     at java.lang.Thread.run(Thread.java:619)
> >                 >
> >
> org.globus.cog.abstraction.impl.file.FileNotFoundException: File not
> >                 >
> >                 found:
> /var/tmp/postproc-20110711-2209-bx2qm0nb/jobs/e/mkoffset-ea7xcuck/LGU/offset-128
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.getFile(FileResourceImpl.java:225)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.local.FileResourceImpl.putFile(FileResourceImpl.java:268)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.file.AbstractFileResource.putFile(AbstractFileResource.java:158)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:314)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:46)
> >                 >     at
> >                 >
> >
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:487)
> >                 >     at java.lang.Thread.run(Thread.java:619)
> >                 > Progress:  time: Mon, 11 Jul 2011 22:16:39 -0500
> >                  Selecting site:389
> >                 > Stage in:16  Active:9  Checking status:1  Finished
> >                 successfully:38
> >                 > Failed but can retry:4
> >                 > Execution failed:
> >                 >     java.lang.NullPointerException
> >                 >     at
> >                 >
> >
> org.griphyn.vdl.mapping.AbstractDataNode.getValue(AbstractDataNode.java:333)
> >                 >     at
> >                 >
> >
> org.griphyn.vdl.karajan.lib.SetFieldValue.log(SetFieldValue.java:71)
> >                 >     at
> >                 >
> >
> org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:38)
> >                 >     at
> >                 >
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:62)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >                 >     at
> >                 >
> >
> org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:66)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> >                 >     at
> >                 >
> >
> org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
> >                 >
> >                 >
> >                 > Any clues?
> >                 >
> >                 > Thanks,
> >                 > --
> >                 > Ketan
> >                 >
> >                 >
> >                 >
> >                 >
> >                 >
> >                 >
> >                 > --
> >                 > Ketan
> >                 >
> >                 >
> >
> >
> >
> >
> >
> >
> >
> >         --
> >         Ketan
> >
> >
> >
> >         _______________________________________________
> >         Swift-devel mailing list
> >         Swift-devel at ci.uchicago.edu
> >
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110714/b8ad0534/attachment.html>

From yadudoc1729 at gmail.com  Thu Jul 14 14:11:55 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 15 Jul 2011 00:41:55 +0530
Subject: [Swift-devel] Updates - Tests
Message-ID: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>

Hi,

I've been fixing failed tests from the test suite after adding changes
to the associative arrays. The support for types as subscripts are
not ready yet, and I will work on those as soon as tests are okayed.

I've run tests and fixed most of the issues I think comes from the
code I've modified. There are some tests which are failing because
of no-permission and file/directory not found errors. I'm not sure
what I should do here. So can can someone check ? I'm working
on an older revision (4697 if I remember correctly).

I'm attaching the logs from the test as well as a patch.

-- 
Thanks and Regards,
Yadu Nand B
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sixth.patch
Type: text/x-patch
Size: 142256 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/a57f23fa/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tests.log
Type: text/x-log
Size: 165622 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/a57f23fa/attachment-0001.bin>

From wilde at mcs.anl.gov  Thu Jul 14 14:18:08 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 14 Jul 2011 14:18:08 -0500 (CDT)
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
Message-ID: <2062654826.96275.1310671088757.JavaMail.root@zimbra.anl.gov>

Yadu, Alberto, Justin, Mihael - can you coordinate on this via the list?

- Mike

----- Original Message -----
> Hi,
> 
> I've been fixing failed tests from the test suite after adding changes
> to the associative arrays. The support for types as subscripts are
> not ready yet, and I will work on those as soon as tests are okayed.
> 
> I've run tests and fixed most of the issues I think comes from the
> code I've modified. There are some tests which are failing because
> of no-permission and file/directory not found errors. I'm not sure
> what I should do here. So can can someone check ? I'm working
> on an older revision (4697 if I remember correctly).
> 
> I'm attaching the logs from the test as well as a patch.
> 
> --
> Thanks and Regards,
> Yadu Nand B
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Thu Jul 14 14:19:44 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 12:19:44 -0700
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
Message-ID: <1310671184.10910.4.camel@blabla>

So the rest of the trunk code seems to pass most (if not all) of the
local tests. Perhaps it's time to integrate your changes into trunk and
go from there. Let me know when you want that to happen, and I'll merge
your patch locally, tidy it up and commit.

If there are things in your code that you feel are hackish (i.e. quick
solutions that seem shady for production), let's discuss/fix them before
we do the above.

On Fri, 2011-07-15 at 00:41 +0530, Yadu Nand wrote:
> Hi,
> 
> I've been fixing failed tests from the test suite after adding changes
> to the associative arrays. The support for types as subscripts are
> not ready yet, and I will work on those as soon as tests are okayed.
> 
> I've run tests and fixed most of the issues I think comes from the
> code I've modified. There are some tests which are failing because
> of no-permission and file/directory not found errors. I'm not sure
> what I should do here. So can can someone check ? I'm working
> on an older revision (4697 if I remember correctly).
> 
> I'm attaching the logs from the test as well as a patch.
> 




From yadudoc1729 at gmail.com  Thu Jul 14 14:42:12 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 15 Jul 2011 01:12:12 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <1310671184.10910.4.camel@blabla>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
Message-ID: <CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>

> So the rest of the trunk code seems to pass most (if not all) of the
> local tests. Perhaps it's time to integrate your changes into trunk and
> go from there. Let me know when you want that to happen, and I'll merge
> your patch locally, tidy it up and commit.

Some of the tests appear to fail, but when I run them independently they
work. I'm not sure what is going on there.

> If there are things in your code that you feel are hackish (i.e. quick
> solutions that seem shady for production), let's discuss/fix them before
> we do the above.

Yes!  In org/griphyn/vdl/engine/Karajan.java, the foreachstat function is
mostly hacks. I'm putting in a code paste :  http://ideone.com/yWbwU


-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Thu Jul 14 15:23:02 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 13:23:02 -0700
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
Message-ID: <1310674982.11160.0.camel@blabla>

On Fri, 2011-07-15 at 01:12 +0530, Yadu Nand wrote:

> > If there are things in your code that you feel are hackish (i.e. quick
> > solutions that seem shady for production), let's discuss/fix them before
> > we do the above.
> 
> Yes!  In org/griphyn/vdl/engine/Karajan.java, the foreachstat function is
> mostly hacks. I'm putting in a code paste :  http://ideone.com/yWbwU
> 

It would be helpful if you were more specific (i.e. you could describe
each individual problem).




From jonmon at utexas.edu  Thu Jul 14 15:30:31 2011
From: jonmon at utexas.edu (Jonathan Monette)
Date: Thu, 14 Jul 2011 15:30:31 -0500
Subject: [Swift-devel] OSG hang
In-Reply-To: <1310670426.9855.2.camel@blabla>
References: <1378282377.95944.1310667776161.JavaMail.root@zimbra.anl.gov>
	<1310667946.6072.7.camel@blabla>
	<4149480F-4287-4778-AC8E-AFF3590B2BC0@utexas.edu>
	<1310668354.6072.10.camel@blabla>
	<ADBED7C4-2D45-4D59-B3EF-4575B4EE1CAF@utexas.edu>
	<1310670426.9855.2.camel@blabla>
Message-ID: <9A7AAEA0-9DC3-4FF5-8B79-34568428D28A@utexas.edu>

I set GLOBUS_HOSTNAME to 50.16.138.64
PING ec2-50-16-138-64.compute-1.amazonaws.com (50.16.138.64): 56 data bytes
Request timeout for icmp_seq 0

jonmon at ip-10-111-45-63:~/sleep$ export GLOBUS_HOSTNAME=50.16.138.64
jonmon at ip-10-111-45-63:~/sleep$ echo $GLOBUS_HOSTNAME
50.16.138.64

I then ran Swift.  I checked the site with:
[jonmon at communicado: ~]$ globus-job-run tuscany.med.harvard.edu /bin/sh -c '/bin/ps -g engage'
  PID TTY          TIME CMD
 3934 ?        00:00:00 globus-job-mana
 4195 ?        00:00:00 perl
 4197 ?        00:00:01 perl
12881 ?        00:00:00 globus-job-mana
13032 ?        00:00:00 perl
17440 ?        00:00:00 condor_shadow
18982 ?        00:00:00 globus-gridftp-
19118 ?        00:00:00 globus-job-mana
19213 ?        00:00:00 bash
19227 ?        00:00:00 sleep
22107 ?        00:00:00 globus-job-mana
22243 ?        00:00:00 ps

sleep did execute on the head node.  sleep finished executing but Swift is still running on the VM.

On Jul 14, 2011, at 2:07 PM, Mihael Hategan wrote:

> On Thu, 2011-07-14 at 14:02 -0500, Jonathan Monette wrote:
> 
>> 
>> So it does seem like the job get run on the head node.  When the job
>> is finished on the head node(I checked several times with
>> globus-job-run) Swift continues to execute(well it just prints the
>> Submitted: 1 line over and over again).
>> 
>> So the next step is to set those environment variables Mike mentioned and try again?  
> 
> Yes. Swift won't work properly if those are not set properly. On most
> system it guesses them, but not on weird systems.
> 
> Setting GLOBUS_HOSTNAME is probably enough though.
> 
> 



From wilde at mcs.anl.gov  Thu Jul 14 16:11:44 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 14 Jul 2011 16:11:44 -0500 (CDT)
Subject: [Swift-devel] OSG hang
In-Reply-To: <9A7AAEA0-9DC3-4FF5-8B79-34568428D28A@utexas.edu>
Message-ID: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>

Jack tells me you cant ping EC2 VMs form outside because they block ICMP.

Check with Jack on what you need to do (if anything) to make sure ports 50000 through 51000 are open, and then make sure GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE are set to 50000,51000

Then make sure (via say the nc or netcat commands) that you can connect to those ports from the OSG site.

If all that works, then work with Mihael to test a simple Karajan script that submits a job.  There are examples of these from Allan in /home/wilde/swift/lab/osg/allantools (or nearby there)

- Mike

----- Original Message -----
> I set GLOBUS_HOSTNAME to 50.16.138.64
> PING ec2-50-16-138-64.compute-1.amazonaws.com (50.16.138.64): 56 data
> bytes
> Request timeout for icmp_seq 0
> 
> jonmon at ip-10-111-45-63:~/sleep$ export GLOBUS_HOSTNAME=50.16.138.64
> jonmon at ip-10-111-45-63:~/sleep$ echo $GLOBUS_HOSTNAME
> 50.16.138.64
> 
> I then ran Swift. I checked the site with:
> [jonmon at communicado: ~]$ globus-job-run tuscany.med.harvard.edu
> /bin/sh -c '/bin/ps -g engage'
> PID TTY TIME CMD
> 3934 ? 00:00:00 globus-job-mana
> 4195 ? 00:00:00 perl
> 4197 ? 00:00:01 perl
> 12881 ? 00:00:00 globus-job-mana
> 13032 ? 00:00:00 perl
> 17440 ? 00:00:00 condor_shadow
> 18982 ? 00:00:00 globus-gridftp-
> 19118 ? 00:00:00 globus-job-mana
> 19213 ? 00:00:00 bash
> 19227 ? 00:00:00 sleep
> 22107 ? 00:00:00 globus-job-mana
> 22243 ? 00:00:00 ps
> 
> sleep did execute on the head node. sleep finished executing but Swift
> is still running on the VM.
> 
> On Jul 14, 2011, at 2:07 PM, Mihael Hategan wrote:
> 
> > On Thu, 2011-07-14 at 14:02 -0500, Jonathan Monette wrote:
> >
> >>
> >> So it does seem like the job get run on the head node. When the job
> >> is finished on the head node(I checked several times with
> >> globus-job-run) Swift continues to execute(well it just prints the
> >> Submitted: 1 line over and over again).
> >>
> >> So the next step is to set those environment variables Mike
> >> mentioned and try again?
> >
> > Yes. Swift won't work properly if those are not set properly. On
> > most
> > system it guesses them, but not on weird systems.
> >
> > Setting GLOBUS_HOSTNAME is probably enough though.
> >
> >
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Thu Jul 14 16:26:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 14:26:34 -0700
Subject: [Swift-devel] OSG hang
In-Reply-To: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>
References: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>
Message-ID: <1310678794.11403.1.camel@blabla>

On Thu, 2011-07-14 at 16:11 -0500, Michael Wilde wrote:
> Jack tells me you cant ping EC2 VMs form outside because they block ICMP.
> 
> Check with Jack on what you need to do (if anything) to make sure ports 50000 through 51000 are open, and then make sure GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE are set to 50000,51000
> 
> Then make sure (via say the nc or netcat commands) that you can connect to those ports from the OSG site.
> 
> If all that works, then work with Mihael to test a simple Karajan
> script that submits a job.  There are examples of these from Allan
> in /home/wilde/swift/lab/osg/allantools (or nearby there)

There's also swift/bin/checksites.k

run it with "swift checksites.k <your_sites.xml>"

You may want to edit it and change the timeout value to something more
than 10s (given that OSG uses managed fork - i.e. fork jobs have a queue
of their own, hence they are not immediate).




From jonmon at mcs.anl.gov  Thu Jul 14 16:47:16 2011
From: jonmon at mcs.anl.gov (Jonathan Monette)
Date: Thu, 14 Jul 2011 16:47:16 -0500
Subject: [Swift-devel] OSG hang
In-Reply-To: <1310678794.11403.1.camel@blabla>
References: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>
	<1310678794.11403.1.camel@blabla>
Message-ID: <4CDC5920-AFDD-43AC-BD3C-F432290C76C7@mcs.anl.gov>

Ok.  I set GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE to 50000,51000 and GLOBUS_HOSTNAME to the ec2 ip-address.  Setting these variables seemed to have resolved this issue.  I can execute Swift on the VM to sites from OSG. I can see the sleep job running on the head node of the OSG site and when it finishes Swift also shuts down.

On Jul 14, 2011, at 4:26 PM, Mihael Hategan wrote:

> On Thu, 2011-07-14 at 16:11 -0500, Michael Wilde wrote:
>> Jack tells me you cant ping EC2 VMs form outside because they block ICMP.
>> 
>> Check with Jack on what you need to do (if anything) to make sure ports 50000 through 51000 are open, and then make sure GLOBUS_TCP_PORT_RANGE and GLOBUS_TCP_SOURCE_RANGE are set to 50000,51000
>> 
>> Then make sure (via say the nc or netcat commands) that you can connect to those ports from the OSG site.
>> 
>> If all that works, then work with Mihael to test a simple Karajan
>> script that submits a job.  There are examples of these from Allan
>> in /home/wilde/swift/lab/osg/allantools (or nearby there)
> 
> There's also swift/bin/checksites.k
> 
> run it with "swift checksites.k <your_sites.xml>"
> 
> You may want to edit it and change the timeout value to something more
> than 10s (given that OSG uses managed fork - i.e. fork jobs have a queue
> of their own, hence they are not immediate).
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From yadudoc1729 at gmail.com  Thu Jul 14 17:00:55 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 15 Jul 2011 03:30:55 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <1310674982.11160.0.camel@blabla>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
Message-ID: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>

On Fri, Jul 15, 2011 at 1:53 AM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> On Fri, 2011-07-15 at 01:12 +0530, Yadu Nand wrote:
>
>> > If there are things in your code that you feel are hackish (i.e. quick
>> > solutions that seem shady for production), let's discuss/fix them before
>> > we do the above.
>>
>> Yes! ?In org/griphyn/vdl/engine/Karajan.java, the foreachstat function is
>> mostly hacks. I'm putting in a code paste : ?http://ideone.com/yWbwU
>>
>
> It would be helpful if you were more specific (i.e. you could describe
> each individual problem).

Ok.
When we state  foreach value, index in array[5]  I'm trying to match the
type of array which from the "in" variable type, something like
int [int] against int [int][string] so that we get the type of the index.
I think we could do this better with regex and here I'm having to extract
the array identifier using very bad string matching. Then we find the
index type and apply it.

That would be between line number 40 and 75 of the code pasted
here -> http://ideone.com/yWbwU

-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Thu Jul 14 17:30:29 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 14 Jul 2011 15:30:29 -0700
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
Message-ID: <1310682629.12137.1.camel@blabla>

On Fri, 2011-07-15 at 03:30 +0530, Yadu Nand wrote:
> On Fri, Jul 15, 2011 at 1:53 AM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > On Fri, 2011-07-15 at 01:12 +0530, Yadu Nand wrote:
> >
> >> > If there are things in your code that you feel are hackish (i.e. quick
> >> > solutions that seem shady for production), let's discuss/fix them before
> >> > we do the above.
> >>
> >> Yes!  In org/griphyn/vdl/engine/Karajan.java, the foreachstat function is
> >> mostly hacks. I'm putting in a code paste :  http://ideone.com/yWbwU
> >>
> >
> > It would be helpful if you were more specific (i.e. you could describe
> > each individual problem).
> 
> Ok.
> When we state  foreach value, index in array[5]  I'm trying to match the
> type of array which from the "in" variable type, something like
> int [int] against int [int][string] so that we get the type of the index.
> I think we could do this better with regex and here I'm having to extract
> the array identifier using very bad string matching. Then we find the
> index type and apply it.
> 
> That would be between line number 40 and 75 of the code pasted
> here -> http://ideone.com/yWbwU
> 

Also the <variable> thing. I have a suspicion that you can use a
function on the template to extract the contents of that tag rather than
rendering it to a string and then parsing it again in custom code.

So let me know if you want to fix those issues. They will have to be
fixed sooner or later by somebody.

Mihael



From benc at hawaga.org.uk  Fri Jul 15 02:25:05 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 15 Jul 2011 07:25:05 +0000 (GMT)
Subject: [Swift-devel] OSG hang
In-Reply-To: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>
References: <1054676944.96854.1310677904530.JavaMail.root@zimbra.anl.gov>
Message-ID: <Pine.LNX.4.64.1107150722160.14908@dildano.hawaga.org.uk>

> Jack tells me you cant ping EC2 VMs form outside because they block 
> ICMP.

I can ping mine.

$ ping s0.barwen.ch
PING s0.barwen.ch (46.51.169.7): 56 data bytes
64 bytes from 46.51.169.7: icmp_seq=0 ttl=50 time=49.800 ms

-- 



From wilde at mcs.anl.gov  Fri Jul 15 08:32:18 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 15 Jul 2011 08:32:18 -0500 (CDT)
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
Message-ID: <1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>

Yadu, I just noticed your link to ideone.com, and realized what that site is about.

Do you already have Swift running there? If not, what's involved in setting that up?
Can runs there run on remote resources?

Ketan mentioned a similar idea a while back (I cant recall what demo site it was) but no one has had time to pursue it.

Also Jon is getting Swift running on Globus Online at the moment; maybe these things could link to each other (eg an ideone.com demo that can also run tiny scripts on GO resources).

This might be nice to brainstorm as you seem to be familiar with ideone.

- Mike


----- Original Message -----
> On Fri, Jul 15, 2011 at 1:53 AM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> > On Fri, 2011-07-15 at 01:12 +0530, Yadu Nand wrote:
> >
> >> > If there are things in your code that you feel are hackish (i.e.
> >> > quick
> >> > solutions that seem shady for production), let's discuss/fix them
> >> > before
> >> > we do the above.
> >>
> >> Yes! In org/griphyn/vdl/engine/Karajan.java, the foreachstat
> >> function is
> >> mostly hacks. I'm putting in a code paste : http://ideone.com/yWbwU
> >>
> >
> > It would be helpful if you were more specific (i.e. you could
> > describe
> > each individual problem).
> 
> Ok.
> When we state foreach value, index in array[5] I'm trying to match the
> type of array which from the "in" variable type, something like
> int [int] against int [int][string] so that we get the type of the
> index.
> I think we could do this better with regex and here I'm having to
> extract
> the array identifier using very bad string matching. Then we find the
> index type and apply it.
> 
> That would be between line number 40 and 75 of the code pasted
> here -> http://ideone.com/yWbwU
> 
> --
> Thanks and Regards,
> Yadu Nand B
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From benc at hawaga.org.uk  Fri Jul 15 08:35:25 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 15 Jul 2011 13:35:25 +0000 (GMT)
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
References: <1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
Message-ID: <Pine.LNX.4.64.1107151334390.14908@dildano.hawaga.org.uk>


> Do you already have Swift running there? If not, what's involved in setting that up?
> Can runs there run on remote resources?
> 
> Ketan mentioned a similar idea a while back (I cant recall what demo 
> site it was) but no one has had time to pursue it.

Something that looks like tryhaskell.org (and relatives) was mentioned 
before.

-- 


From wilde at mcs.anl.gov  Fri Jul 15 08:39:54 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 15 Jul 2011 08:39:54 -0500 (CDT)
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <Pine.LNX.4.64.1107151334390.14908@dildano.hawaga.org.uk>
Message-ID: <1458597750.97885.1310737194425.JavaMail.root@zimbra.anl.gov>

I just noticed this at ideone.com:

"Can I write or read files in my program?
- No, only stdin and stdout are allowed."

Maybe build something like it into the Swift web based on ideone and tryhaskell etc. ? 

Building a safe but sufficiently interesting sandbox seems the main challenge.

- Mike


----- Original Message -----
> > Do you already have Swift running there? If not, what's involved in
> > setting that up?
> > Can runs there run on remote resources?
> >
> > Ketan mentioned a similar idea a while back (I cant recall what demo
> > site it was) but no one has had time to pursue it.
> 
> Something that looks like tryhaskell.org (and relatives) was mentioned
> before.
> 
> --

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From yadudoc1729 at gmail.com  Fri Jul 15 08:59:06 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Fri, 15 Jul 2011 19:29:06 +0530
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
References: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
Message-ID: <CANa904mVN=OHYoiK=WoWNpcKQXaG1MfGwnW7C23dTGzcG3Paew@mail.gmail.com>

> Do you already have Swift running there? If not, what's involved in setting that up?
> Can runs there run on remote resources?

I haven't tried running swift on it. As far as I understand it is used
only for tiny code
snippets and swift needs much more. If we could have it running on ideone.com it
would have been pretty neat, with easy access from even handhelds.

-- 
Thanks and Regards,
Yadu Nand B


From ketancmaheshwari at gmail.com  Fri Jul 15 09:05:15 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Fri, 15 Jul 2011 09:05:15 -0500
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
References: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
Message-ID: <CAMUuviqxx88MMDO+8ntJzRs9Q70A=j2JiHa7D0CfPmSBW=4MoQ@mail.gmail.com>

On Fri, Jul 15, 2011 at 8:32 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Yadu, I just noticed your link to ideone.com, and realized what that site
> is about.
>
> Do you already have Swift running there? If not, what's involved in setting
> that up?
> Can runs there run on remote resources?
>
> Ketan mentioned a similar idea a while back (I cant recall what demo site
> it was) but no one has had time to pursue it.
>

Mike, I mentioned you this same site (ideone.com) in one of our Swift
meetings. And we also had a brief discussion on list to have something on
the lines of tryruby.org and tryhaskell.org


>
> Also Jon is getting Swift running on Globus Online at the moment; maybe
> these things could link to each other (eg an ideone.com demo that can also
> run tiny scripts on GO resources).
>
> This might be nice to brainstorm as you seem to be familiar with ideone.
>
> - Mike
>
>
> ----- Original Message -----
> > On Fri, Jul 15, 2011 at 1:53 AM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> > > On Fri, 2011-07-15 at 01:12 +0530, Yadu Nand wrote:
> > >
> > >> > If there are things in your code that you feel are hackish (i.e.
> > >> > quick
> > >> > solutions that seem shady for production), let's discuss/fix them
> > >> > before
> > >> > we do the above.
> > >>
> > >> Yes! In org/griphyn/vdl/engine/Karajan.java, the foreachstat
> > >> function is
> > >> mostly hacks. I'm putting in a code paste : http://ideone.com/yWbwU
> > >>
> > >
> > > It would be helpful if you were more specific (i.e. you could
> > > describe
> > > each individual problem).
> >
> > Ok.
> > When we state foreach value, index in array[5] I'm trying to match the
> > type of array which from the "in" variable type, something like
> > int [int] against int [int][string] so that we get the type of the
> > index.
> > I think we could do this better with regex and here I'm having to
> > extract
> > the array identifier using very bad string matching. Then we find the
> > index type and apply it.
> >
> > That would be between line number 40 and 75 of the code pasted
> > here -> http://ideone.com/yWbwU
> >
> > --
> > Thanks and Regards,
> > Yadu Nand B
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/c99f89be/attachment.html>

From ketancmaheshwari at gmail.com  Fri Jul 15 09:08:40 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Fri, 15 Jul 2011 09:08:40 -0500
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <CANa904mVN=OHYoiK=WoWNpcKQXaG1MfGwnW7C23dTGzcG3Paew@mail.gmail.com>
References: <CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1388966217.97839.1310736738087.JavaMail.root@zimbra.anl.gov>
	<CANa904mVN=OHYoiK=WoWNpcKQXaG1MfGwnW7C23dTGzcG3Paew@mail.gmail.com>
Message-ID: <CAMUuviorJH0cj+wS6M0oZdt-NEN1m+CS1J2U4QP6n4W-hDx38A@mail.gmail.com>

According to the site, if we can build a gentoo package for Swift we can add
it on the site and should work.

On Fri, Jul 15, 2011 at 8:59 AM, Yadu Nand <yadudoc1729 at gmail.com> wrote:

> > Do you already have Swift running there? If not, what's involved in
> setting that up?
> > Can runs there run on remote resources?
>
> I haven't tried running swift on it. As far as I understand it is used
> only for tiny code
> snippets and swift needs much more. If we could have it running on
> ideone.com it
> would have been pretty neat, with easy access from even handhelds.
>
> --
> Thanks and Regards,
> Yadu Nand B
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/1a79c6c9/attachment.html>

From ketancmaheshwari at gmail.com  Fri Jul 15 09:11:18 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Fri, 15 Jul 2011 09:11:18 -0500
Subject: [Swift-devel] Swift demos on ideone.com or similar?
In-Reply-To: <1458597750.97885.1310737194425.JavaMail.root@zimbra.anl.gov>
References: <Pine.LNX.4.64.1107151334390.14908@dildano.hawaga.org.uk>
	<1458597750.97885.1310737194425.JavaMail.root@zimbra.anl.gov>
Message-ID: <CAMUuvipNDKpfYLj74HQ5FLff14jU+WzjfSF0b51dC1K2cDZapQ@mail.gmail.com>

I also have the domain booked if we want to go for it:

http://tryswift.org/  :)



On Fri, Jul 15, 2011 at 8:39 AM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> I just noticed this at ideone.com:
>
> "Can I write or read files in my program?
> - No, only stdin and stdout are allowed."
>
> Maybe build something like it into the Swift web based on ideone and
> tryhaskell etc. ?
>
> Building a safe but sufficiently interesting sandbox seems the main
> challenge.
>
> - Mike
>
>
> ----- Original Message -----
> > > Do you already have Swift running there? If not, what's involved in
> > > setting that up?
> > > Can runs there run on remote resources?
> > >
> > > Ketan mentioned a similar idea a while back (I cant recall what demo
> > > site it was) but no one has had time to pursue it.
> >
> > Something that looks like tryhaskell.org (and relatives) was mentioned
> > before.
> >
> > --
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/8df4d281/attachment.html>

From yadudoc1729 at gmail.com  Fri Jul 15 13:33:46 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Sat, 16 Jul 2011 00:03:46 +0530
Subject: [Swift-devel] Cleaning up the code inside foreachstat( )
Message-ID: <CANa904mt-pymsgJikE2ZoB6nUxC8UmbADqa7F076_W1qxaqKEQ@mail.gmail.com>

Hi,

During the regular meeting with Justin, we discussed cleaning up
the code inside foreachstat (Karajan.java). Justin suggested
placing code for extracting the string which is the array identifier
within the Foreach class. I see that Foreach from Foreach.java
is an interface, and one that is auto-generated. I also see the
banner saying *Automatically generated - do not modify.*
ForeachImpl which contains code also has the same no modify
banner.

Now I'm in a fix, what should I do ? I could just take the dirty looking
code and place it a function extractArrayIdentifier( ).

-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Fri Jul 15 13:42:22 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 15 Jul 2011 11:42:22 -0700
Subject: [Swift-devel] Cleaning up the code inside foreachstat( )
In-Reply-To: <CANa904mt-pymsgJikE2ZoB6nUxC8UmbADqa7F076_W1qxaqKEQ@mail.gmail.com>
References: <CANa904mt-pymsgJikE2ZoB6nUxC8UmbADqa7F076_W1qxaqKEQ@mail.gmail.com>
Message-ID: <1310755342.16525.0.camel@blabla>

On Sat, 2011-07-16 at 00:03 +0530, Yadu Nand wrote:
> Hi,
> 
> During the regular meeting with Justin, we discussed cleaning up
> the code inside foreachstat (Karajan.java). Justin suggested
> placing code for extracting the string which is the array identifier
> within the Foreach class. I see that Foreach from Foreach.java
> is an interface, and one that is auto-generated. I also see the
> banner saying *Automatically generated - do not modify.*
> ForeachImpl which contains code also has the same no modify
> banner.
> 
> Now I'm in a fix, what should I do ? I could just take the dirty looking
> code and place it a function extractArrayIdentifier( ).
> 

That sounds reasonable.




From alberto_chavez at live.com  Fri Jul 15 14:24:12 2011
From: alberto_chavez at live.com (Alberto Chavez)
Date: Fri, 15 Jul 2011 14:24:12 -0500
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <2062654826.96275.1310671088757.JavaMail.root@zimbra.anl.gov>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>,
	<2062654826.96275.1310671088757.JavaMail.root@zimbra.anl.gov>
Message-ID: <BAY156-w23F1D351A0CCA86D2F29FB9C490@phx.gbl>



> Yadu, Alberto, Justin, Mihael - can you coordinate on this via the list?
Hi All,I am sorry for my absence, as some of you already know, my father died last week and I am just trying to catch-up with the team. I've noticed a bunch of changes on the test suite, and I would like to know what things needs to be done, or what should I start doing.
Thank you,
Alberto. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110715/77135aa5/attachment.html>

From iraicu at cs.iit.edu  Sun Jul 17 04:21:48 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sun, 17 Jul 2011 04:21:48 -0500
Subject: [Swift-devel] deadline extension 08/16/11 -- SI on Data Intensive
 Computing in the Clouds in JGC
Message-ID: <4E22A9AC.2060504@cs.iit.edu>

Hi all,
Due to numerous requests, we have extended the deadline for the Special 
Issue on Data Intensive Computing in the Clouds in the Springer Journal 
of Grid Computing to August 16th, 2011. Please see below for the CFP 
announcement.

Regards,
Ioan Raicu and Tevfik Kosar
Guest Editors of the Special Issue on Data Intensive Computing in the Clouds
Springer Journal of Grid Computing
http://datasys.cs.iit.edu/events/JGC-DataCloud-2012/index.html

---------------------------------------------------------------------------------
			    *** Call for Papers ***					
		      Springer Journal of Grid Computing
	    Special Issue on Data Intensive Computing in the Clouds
	     http://datasys.cs.iit.edu/events/JGC-DataCloud-2012/
---------------------------------------------------------------------------------

Applications and experiments in all areas of science are becoming increasingly
complex and more demanding in terms of their computational and data requirements.
Some applications generate data volumes reaching hundreds of terabytes and even
petabytes. As scientific applications become more data intensive, the management
of data resources and dataflow between the storage and compute resources is
becoming the main bottleneck. Analyzing, visualizing, and disseminating these
large data sets has become a major challenge and data intensive computing is now
considered as the "fourth paradigm" in scientific discovery after
empirical, theoretical, and computational scientific approaches.

The Special Issue on Data Intensive Computing in the Clouds will provide the
scientific community a dedicated forum, within the prestigious Springer Journal
of Grid Computing, for presenting new research, development, and deployment
efforts in running data-intensive computing workloads on Cloud Computing
infrastructures. This special issue will focus on the use of cloud-based
technologies to meet the new data intensive scientific challenges that are not
well served by the current supercomputers, grids or compute-intensive clouds.
We believe this venue will be an excellent place to help the community define the
current state, determine future goals, and present architectures and services for
future clouds supporting data intensive computing.


TOPICS
---------------------------------------------------------------------------------
- Data-intensive cloud computing applications, characteristics, challenges
- Case studies of data intensive computing in the clouds
- Performance evaluation of data clouds, data grids, and data centers
- Energy-efficient data cloud design and management
- Data placement, scheduling, and interoperability in the clouds
- Accountability, QoS, and SLAs
- Data privacy and protection in a public cloud environment
- Distributed file systems for clouds
- Data streaming and parallelization
- New programming models for data-intensive cloud computing
- Scalability issues in clouds
- Social computing and massively social gaming
- 3D Internet and implications
- Future research challenges in data-intensive cloud computing

Important Dates
---------------------------------------------------------------------------------------
*	Papers Due:			August 16, 2011
*	First Round Decisions:		October 15, 2011
*	Major Revisions if needed:	November 15, 2011
*	Second Round Decisions:		December 15, 2011
*	Minor Revisions if needed:	January 15, 2012
*	Final Decision:			February 1, 2012
*	Publication Date:		June 2012

PAPER SUBMISSION
---------------------------------------------------------------------------------
Authors are invited to submit original and unpublished technical papers. All
submissions will be peer-reviewed and judged on correctness, originality,
technical strength, significance, quality of presentation, and relevance to the
special issue topics of interest. Submitted papers may not have	appeared in or be
under consideration for another workshop, conference or a journal, nor may they
be under review or submitted to another forum during the review process.
Submitted papers may not exceed 20 single-spaced double-column pages using
10-point size font on 8.5x11 inch pages (1" margins), including figures, tables,
and references; note that accepted papers will likely be between 15 to 20 pages,
depending on a variety of factors; for more information for preparing the
submitted papers, please see
http://www.springer.com/computer/communication+networks/journal/10723, under
"Instructions for Authors". The papers (PDF format) must be submitted online at
http://grid.edmgr.com/ before the extended deadline of August 16th, 2011
at 11:59PM PST. For any questions on the submission process, please email the
guest editors at jgc-datacloud-2012 at datasys.cs.iit.edu.

Guest Editors
---------------------------------------------------------------------------------------
Special Issue Guest Editors
*	Tevfik Kosar (tkosar at buffalo.edu), University at Buffalo
*	Ioan Raicu (iraicu at cs.iit.edu), Illinois Institute of Technology&
					Argonne National Laboratory

Editors-in-Chief
*	Peter Kacsuk, Hungarian Academy of Sciences
*	Ian Foster, University of Chicago&  Argonne National Laboratory



-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================




From dsk at ci.uchicago.edu  Sun Jul 17 12:02:12 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Sun, 17 Jul 2011 11:02:12 -0600
Subject: [Swift-devel] pilot job paper
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
Message-ID: <0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>

This paper, which is under review (I'm not sure for which conference) compares Coasters with some other pilot job mechanisms.  I haven't yet read it, so I don't know how accurate or useful it is, but the concept is interesting.

Dan


Begin forwarded message:

> From: Shantenu Jha <sjha at cct.lsu.edu>
> Date: July 17, 2011 10:41:57 AM MDT
> To: "Daniel S. Katz" <dsk at ci.uchicago.edu>
> Subject: URL
> 
> 
> http://www.cct.lsu.edu/~sjha/select_publications/pstar.pdf

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/7b7b0923/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 17 13:22:53 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 11:22:53 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
Message-ID: <1310926973.31901.2.camel@blabla>

It seems to miss that Swift is not the only end-user interface for
coasters or that resources can be managed dynamically through the API
(i.e. sites.xml is specific to the Swift layer).

We (I that is) really need to finish that coaster paper.

On Sun, 2011-07-17 at 11:02 -0600, Daniel S. Katz wrote:
> This paper, which is under review (I'm not sure for which conference)
> compares Coasters with some other pilot job mechanisms.  I haven't yet
> read it, so I don't know how accurate or useful it is, but the concept
> is interesting.
> 
> 
> Dan
> 
> 
> Begin forwarded message:
> 
> > From: Shantenu Jha <sjha at cct.lsu.edu>
> > 
> > Date: July 17, 2011 10:41:57 AM MDT
> > 
> > To: "Daniel S. Katz" <dsk at ci.uchicago.edu>
> > 
> > Subject: URL
> > 
> > 
> > 
> > http://www.cct.lsu.edu/~sjha/select_publications/pstar.pdf
> > 
> 
> -- 
> Daniel S. Katz
> University of Chicago
> (773) 834-7186 (voice)
> (773) 834-6818 (fax)
> d.katz at ieee.org or dsk at ci.uchicago.edu
> http://www.ci.uchicago.edu/~dsk/
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From wilde at mcs.anl.gov  Sun Jul 17 13:52:24 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 17 Jul 2011 13:52:24 -0500 (CDT)
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310926973.31901.2.camel@blabla>
Message-ID: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>



----- Original Message -----
> It seems to miss that Swift is not the only end-user interface for
> coasters or that resources can be managed dynamically through the API
> (i.e. sites.xml is specific to the Swift layer).
> 
> We (I that is) really need to finish that coaster paper.

Theres a special issue of J. Grid Computing call for data-intensive Cloud papers due Aug 16 that might be good for this (perhaps stressing the provide-staging aspects)?

- Mike

> 
> On Sun, 2011-07-17 at 11:02 -0600, Daniel S. Katz wrote:
> > This paper, which is under review (I'm not sure for which
> > conference)
> > compares Coasters with some other pilot job mechanisms. I haven't
> > yet
> > read it, so I don't know how accurate or useful it is, but the
> > concept
> > is interesting.
> >
> >
> > Dan
> >
> >
> > Begin forwarded message:
> >
> > > From: Shantenu Jha <sjha at cct.lsu.edu>
> > >
> > > Date: July 17, 2011 10:41:57 AM MDT
> > >
> > > To: "Daniel S. Katz" <dsk at ci.uchicago.edu>
> > >
> > > Subject: URL
> > >
> > >
> > >
> > > http://www.cct.lsu.edu/~sjha/select_publications/pstar.pdf
> > >
> >
> > --
> > Daniel S. Katz
> > University of Chicago
> > (773) 834-7186 (voice)
> > (773) 834-6818 (fax)
> > d.katz at ieee.org or dsk at ci.uchicago.edu
> > http://www.ci.uchicago.edu/~dsk/
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Sun Jul 17 13:59:45 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 11:59:45 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
Message-ID: <1310929185.32625.1.camel@blabla>

On Sun, 2011-07-17 at 13:52 -0500, Michael Wilde wrote:
> 
> ----- Original Message -----
> > It seems to miss that Swift is not the only end-user interface for
> > coasters or that resources can be managed dynamically through the API
> > (i.e. sites.xml is specific to the Swift layer).
> > 
> > We (I that is) really need to finish that coaster paper.
> 
> Theres a special issue of J. Grid Computing call for data-intensive
> Cloud papers due Aug 16 that might be good for this (perhaps stressing
> the provide-staging aspects)?

Sounds good. I'll start getting some of the necessary numbers.




From hategan at mcs.anl.gov  Sun Jul 17 14:07:32 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 12:07:32 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310929185.32625.1.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla>
Message-ID: <1310929652.32736.1.camel@blabla>

On Sun, 2011-07-17 at 11:59 -0700, Mihael Hategan wrote:
> > 
> > Theres a special issue of J. Grid Computing call for data-intensive
> > Cloud papers due Aug 16 that might be good for this (perhaps stressing
> > the provide-staging aspects)?
> 
> Sounds good. I'll start getting some of the necessary numbers.

Which reminds me: who wants to help with said numbers?

For example, there is an easy one: the dependence between job wall time
and queuing time on various clusters.



From dsk at ci.uchicago.edu  Sun Jul 17 15:18:35 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Sun, 17 Jul 2011 14:18:35 -0600
Subject: [Swift-devel] Swift and FutureGRid
Message-ID: <099ABFEC-6A5D-4EA8-B331-DB3D9DA2AC2B@ci.uchicago.edu>

FutureGrid is looking for projects to install software that can be used by lots of users - perhaps we should consider installing Swift there.

Dan


-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/112e9df5/attachment.html>

From wilde at mcs.anl.gov  Sun Jul 17 15:41:16 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sun, 17 Jul 2011 15:41:16 -0500 (CDT)
Subject: [Swift-devel] Swift and FutureGRid
In-Reply-To: <099ABFEC-6A5D-4EA8-B331-DB3D9DA2AC2B@ci.uchicago.edu>
Message-ID: <1734413374.101416.1310935276534.JavaMail.root@zimbra.anl.gov>

Done :) David's been running Swift there in collaboration with Kate and John on Nimbus. But yes, lets get this rolling and get some apps and users using FG. David, can you forward your note to the list on your progress and next steps? 


- Mike 

----- Original Message -----


FutureGrid is looking for projects to install software that can be used by lots of users - perhaps we should consider installing Swift there. 


Dan 







-- 








Daniel S. Katz 
University of Chicago 
(773) 834-7186 (voice) 
(773) 834-6818 (fax) 
d.katz at ieee.org or dsk at ci.uchicago.edu 
http://www.ci.uchicago.edu/~dsk/ 





_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/8bc6e37c/attachment.html>

From ketancmaheshwari at gmail.com  Sun Jul 17 15:59:09 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sun, 17 Jul 2011 15:59:09 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
Message-ID: <CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>

In section IV.C I did not understand this about Swift Coasters:

"" there is no possibility to manage this resource pool programmatically ""

Which seem to be slightly contradicting with the last statement in the same
para:

""SWIFT and Coaster supports various scheduling mechanisms, e. g. a FIFO and
a load-aware scheduler.""

Also, Nimrod/G is mentioned as a pilot-based framework, but no details are
provided. As of 2008, when I used Nimrod/G, it did not have a pilot-based
execution framework.

A bit surprised that MOTEUR's interface with pilot-based system DIANE and
its many interesting recent results are not discussed provided that Marc
Santcroos is one of the developers.

--
Ketan


On Sun, Jul 17, 2011 at 12:02 PM, Daniel S. Katz <dsk at ci.uchicago.edu>wrote:

> This paper, which is under review (I'm not sure for which conference)
> compares Coasters with some other pilot job mechanisms.  I haven't yet read
> it, so I don't know how accurate or useful it is, but the concept is
> interesting.
>
> Dan
>
>
> Begin forwarded message:
>
> *From: *Shantenu Jha <sjha at cct.lsu.edu>
> *Date: *July 17, 2011 10:41:57 AM MDT
> *To: *"Daniel S. Katz" <dsk at ci.uchicago.edu>
> *Subject: **URL*
>
>
> http://www.cct.lsu.edu/~sjha/select_publications/pstar.pdf
>
>
>  --
> Daniel S. Katz
> University of Chicago
> (773) 834-7186 (voice)
> (773) 834-6818 (fax)
> d.katz at ieee.org or dsk at ci.uchicago.edu
> http://www.ci.uchicago.edu/~dsk/
>
>
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/a60f237b/attachment.html>

From benc at hawaga.org.uk  Sun Jul 17 16:04:06 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 17 Jul 2011 21:04:06 +0000 (GMT)
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1107172059590.14908@dildano.hawaga.org.uk>


> Also, Nimrod/G is mentioned as a pilot-based framework, but no details 
> are provided. As of 2008, when I used Nimrod/G, it did not have a 
> pilot-based execution framework.

Nimrod had a pilot-based framework when I visited them at the end of 2009.

I presume that it is used in the /G portion of nimrod (a grid environment 
is one of the places it makes the most sense, I think), but I don't 
remember really.

They were grumbling about how hard it was to make one that works right, 
and seemed fairly pleased that everyone has that trouble ;) (and also 
interested in seeing if they could use something else - falkon at the time 
- in order to avoid putting themselves to the trouble)

-- 


From ketancmaheshwari at gmail.com  Sun Jul 17 16:10:51 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sun, 17 Jul 2011 16:10:51 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310929652.32736.1.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
Message-ID: <CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>

On Sun, Jul 17, 2011 at 2:07 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-07-17 at 11:59 -0700, Mihael Hategan wrote:
> > >
> > > Theres a special issue of J. Grid Computing call for data-intensive
> > > Cloud papers due Aug 16 that might be good for this (perhaps stressing
> > > the provide-staging aspects)?
> >
> > Sounds good. I'll start getting some of the necessary numbers.
>
> Which reminds me: who wants to help with said numbers?
>
> For example, there is an easy one: the dependence between job wall time
> and queuing time on various clusters.
>

I can try this along with the cybershake work I am doing these days.

I read the text of the paper (in March) and have some comments, will send'em
soon.


> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>

-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/566c4750/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 17 16:11:20 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 14:11:20 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
Message-ID: <1310937080.1163.8.camel@blabla>

On Sun, 2011-07-17 at 15:59 -0500, Ketan Maheshwari wrote:
> In section IV.C I did not understand this about Swift Coasters: 
> 
> "" there is no possibility to manage this resource pool
> programmatically ""

That statement is silly by construction. The resource pool is managed
programmatically. It's not like coasters are implemented in a mechanical
device with gears, levers, pulleys, and belts.
> 
> Which seem to be slightly contradicting with the last statement in the
> same para:
> 
> ""SWIFT and Coaster supports various scheduling mechanisms, e. g. a
> FIFO and a load-aware scheduler.""

Swift is separate from coasters. The fifo and load aware scheduler are
from swift. The coasters have the block allocation scheme and its
respective scheduler.
> 
> Also, Nimrod/G is mentioned as a pilot-based framework, but no details
> are provided. As of 2008, when I used Nimrod/G, it did not have a
> pilot-based execution framework.
> 
> A bit surprised that MOTEUR's interface with pilot-based system DIANE
> and its many interesting recent results are not discussed provided
> that Marc Santcroos is one of the developers.

They mention DIANE I thought.




From hategan at mcs.anl.gov  Sun Jul 17 16:14:28 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 14:14:28 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
Message-ID: <1310937268.1163.11.camel@blabla>

On Sun, 2011-07-17 at 16:10 -0500, Ketan Maheshwari wrote:
> 
> 
> On Sun, Jul 17, 2011 at 2:07 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Sun, 2011-07-17 at 11:59 -0700, Mihael Hategan wrote:
>         > >
>         > > Theres a special issue of J. Grid Computing call for
>         data-intensive
>         > > Cloud papers due Aug 16 that might be good for this
>         (perhaps stressing
>         > > the provide-staging aspects)?
>         >
>         > Sounds good. I'll start getting some of the necessary
>         numbers.
>         
>         
>         Which reminds me: who wants to help with said numbers?
>         
>         For example, there is an easy one: the dependence between job
>         wall time
>         and queuing time on various clusters.
> 
> I can try this along with the cybershake work I am doing these days. 

I'm not sure where cybershake fits in. The problem would be to submit
dummy jobs of various walltimes to various clusters and record the
amount of time they sit in the queue.

Though an alternative may be to go through swift logs and extract that
data. It may not give us relevant data points, but it may be worth a
try.
> 
> I read the text of the paper (in March) and have some comments, will
> send'em soon. 
> 
Thanks.




From ketancmaheshwari at gmail.com  Sun Jul 17 16:14:37 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sun, 17 Jul 2011 16:14:37 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310937080.1163.8.camel@blabla>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
	<1310937080.1163.8.camel@blabla>
Message-ID: <CAMUuviqcJTM4s5ubD1n4gyYkO+mbHAkXY-dbBb8EZzfizjpiVA@mail.gmail.com>

> They mention DIANE I thought.
>
>
> Yes they did, but using DIANE without some kind of jobs orchestration with
sufficient job buffer on piloted sites makes little sense and is too
tedious. A workflow engine brings that to the table. MOTEUR was one such
early example of a workflow engine coupled with pilot based framework.

-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/00a08d46/attachment.html>

From ketancmaheshwari at gmail.com  Sun Jul 17 16:19:21 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sun, 17 Jul 2011 16:19:21 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310937268.1163.11.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
Message-ID: <CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>

On Sun, Jul 17, 2011 at 4:14 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-07-17 at 16:10 -0500, Ketan Maheshwari wrote:
> >
> >
> > On Sun, Jul 17, 2011 at 2:07 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >         On Sun, 2011-07-17 at 11:59 -0700, Mihael Hategan wrote:
> >         > >
> >         > > Theres a special issue of J. Grid Computing call for
> >         data-intensive
> >         > > Cloud papers due Aug 16 that might be good for this
> >         (perhaps stressing
> >         > > the provide-staging aspects)?
> >         >
> >         > Sounds good. I'll start getting some of the necessary
> >         numbers.
> >
> >
> >         Which reminds me: who wants to help with said numbers?
> >
> >         For example, there is an easy one: the dependence between job
> >         wall time
> >         and queuing time on various clusters.
> >
> > I can try this along with the cybershake work I am doing these days.
>
> I'm not sure where cybershake fits in. The problem would be to submit
> dummy jobs of various walltimes to various clusters and record the
> amount of time they sit in the queue.
>

It is just a use-case. I thought, stats based on an (any) real application
will have more merit compared to dummy jobs.


> Though an alternative may be to go through swift logs and extract that
> data. It may not give us relevant data points, but it may be worth a
> try.
> >
> > I read the text of the paper (in March) and have some comments, will
> > send'em soon.
> >
> Thanks.
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/2521d7b6/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 17 16:19:53 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 14:19:53 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuviqcJTM4s5ubD1n4gyYkO+mbHAkXY-dbBb8EZzfizjpiVA@mail.gmail.com>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
	<1310937080.1163.8.camel@blabla>
	<CAMUuviqcJTM4s5ubD1n4gyYkO+mbHAkXY-dbBb8EZzfizjpiVA@mail.gmail.com>
Message-ID: <1310937593.1346.0.camel@blabla>

On Sun, 2011-07-17 at 16:14 -0500, Ketan Maheshwari wrote:
> 
>         They mention DIANE I thought.
>         
>         
> Yes they did, but using DIANE without some kind of jobs orchestration
> with sufficient job buffer on piloted sites makes little sense and is
> too tedious. A workflow engine brings that to the table. MOTEUR was
> one such early example of a workflow engine coupled with pilot based
> framework.

I think this is generally the idea with pilot jobs. It makes little
sense to use them for a single job.





From hategan at mcs.anl.gov  Sun Jul 17 16:23:33 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 14:23:33 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
Message-ID: <1310937813.1346.4.camel@blabla>

On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:

>         
>         I'm not sure where cybershake fits in. The problem would be to
>         submit
>         dummy jobs of various walltimes to various clusters and record
>         the
>         amount of time they sit in the queue.
>  
> It is just a use-case. I thought, stats based on an (any) real
> application will have more merit compared to dummy jobs.
> 
I don't see how it would in this case. We're trying to measure queuing
time vs walltime in a general sense. I think it makes more sense to not
have a specific application, which would simply be irrelevant
information.




From dsk at ci.uchicago.edu  Sun Jul 17 16:30:41 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Sun, 17 Jul 2011 15:30:41 -0600
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310937813.1346.4.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
Message-ID: <9D81F54B-1123-4444-866C-84E3D7E14870@ci.uchicago.edu>

I agree for this paper.

Dan


On Jul 17, 2011, at 3:23 PM, Mihael Hategan wrote:

> On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:
> 
>> 
>>        I'm not sure where cybershake fits in. The problem would be to
>>        submit
>>        dummy jobs of various walltimes to various clusters and record
>>        the
>>        amount of time they sit in the queue.
>> 
>> It is just a use-case. I thought, stats based on an (any) real
>> application will have more merit compared to dummy jobs.
>> 
> I don't see how it would in this case. We're trying to measure queuing
> time vs walltime in a general sense. I think it makes more sense to not
> have a specific application, which would simply be irrelevant
> information.
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/






From dsk at ci.uchicago.edu  Sun Jul 17 17:05:49 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Sun, 17 Jul 2011 16:05:49 -0600
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310937080.1163.8.camel@blabla>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
	<1310937080.1163.8.camel@blabla>
Message-ID: <EBE57B95-9ADD-4E3F-B72E-2A8F148A7E5E@ci.uchicago.edu>


On Jul 17, 2011, at 3:11 PM, Mihael Hategan wrote:

> On Sun, 2011-07-17 at 15:59 -0500, Ketan Maheshwari wrote:
>> In section IV.C I did not understand this about Swift Coasters: 
>> 
>> "" there is no possibility to manage this resource pool
>> programmatically ""
> 
> That statement is silly by construction. The resource pool is managed
> programmatically. It's not like coasters are implemented in a mechanical
> device with gears, levers, pulleys, and belts.

I think the point here was that the program that creates the pilot job cannot manage it - the spawner spawns and then goes away.

A couple of the authors are German, and occasionally write in a way that doesn't sound clear in English.

But I'm just guessing - the reviewers might comment on this too...

>> 
>> Which seem to be slightly contradicting with the last statement in the
>> same para:
>> 
>> ""SWIFT and Coaster supports various scheduling mechanisms, e. g. a
>> FIFO and a load-aware scheduler.""
> 
> Swift is separate from coasters. The fifo and load aware scheduler are
> from swift. The coasters have the block allocation scheme and its
> respective scheduler.

Right, which is why these two sentences are not contradictory, I think.

>> 
>> Also, Nimrod/G is mentioned as a pilot-based framework, but no details
>> are provided. As of 2008, when I used Nimrod/G, it did not have a
>> pilot-based execution framework.
>> 
>> A bit surprised that MOTEUR's interface with pilot-based system DIANE
>> and its many interesting recent results are not discussed provided
>> that Marc Santcroos is one of the developers.
> 
> They mention DIANE I thought.
> 
> 

-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/






From hategan at mcs.anl.gov  Sun Jul 17 17:52:54 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 15:52:54 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <EBE57B95-9ADD-4E3F-B72E-2A8F148A7E5E@ci.uchicago.edu>
References: <alpine.LFD.2.00.1107171141490.1391@localhost.localdomain>
	<0485EDE9-ED2D-43D5-A549-0BCF032E6DAA@ci.uchicago.edu>
	<CAMUuviqiYwGiCzZhXW10hR9JE5Tmu0e1uJMar40vRmGgWhP42w@mail.gmail.com>
	<1310937080.1163.8.camel@blabla>
	<EBE57B95-9ADD-4E3F-B72E-2A8F148A7E5E@ci.uchicago.edu>
Message-ID: <1310943174.1703.18.camel@blabla>

On Sun, 2011-07-17 at 16:05 -0600, Daniel S. Katz wrote:
> On Jul 17, 2011, at 3:11 PM, Mihael Hategan wrote:
> 
> > On Sun, 2011-07-17 at 15:59 -0500, Ketan Maheshwari wrote:
> >> In section IV.C I did not understand this about Swift Coasters: 
> >> 
> >> "" there is no possibility to manage this resource pool
> >> programmatically ""
> > 
> > That statement is silly by construction. The resource pool is managed
> > programmatically. It's not like coasters are implemented in a mechanical
> > device with gears, levers, pulleys, and belts.
> 
> I think the point here was that the program that creates the pilot job cannot manage it - the spawner spawns and then goes away.

I think the problem is that 80% of that section is about Swift, and then
the statement is correct if "programmatically" is replaced by
"dynamically" in the context of Swift + any other PJ system.

But in general I dislike "you cannot do x with y" types of statements.
They don't distinguish between a theoretical impossibility, a feature
not yet there (with various levels of implementation difficulty), and
some feature that doesn't make sense for a given system. 

And when somebody says "you cannot programmatically do x with y" that
hints towards a theoretical impossibility, which is never the case if
some other system supports that x.



From ketancmaheshwari at gmail.com  Sun Jul 17 19:53:50 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Sun, 17 Jul 2011 19:53:50 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310937813.1346.4.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
Message-ID: <CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>

On Sun, Jul 17, 2011 at 4:23 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:
>
> >
> >         I'm not sure where cybershake fits in. The problem would be to
> >         submit
> >         dummy jobs of various walltimes to various clusters and record
> >         the
> >         amount of time they sit in the queue.
> >
> > It is just a use-case. I thought, stats based on an (any) real
> > application will have more merit compared to dummy jobs.
> >
> I don't see how it would in this case. We're trying to measure queuing
> time vs walltime in a general sense. I think it makes more sense to not
> have a specific application, which would simply be irrelevant
> information.
>

Ok, we can use bionimbus cloud with Mike's catsnsleep script with following
possible variations:

Job Completion times: sleep parameter 1s-100s
Data : cat file size 1M-1G
Bionimbus VMs 1-32
Number of jobs n, 1-1000

-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110717/316959bf/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 17 20:04:16 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 17 Jul 2011 18:04:16 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
Message-ID: <1310951056.7468.7.camel@blabla>

On Sun, 2011-07-17 at 19:53 -0500, Ketan Maheshwari wrote:
> 
> On Sun, Jul 17, 2011 at 4:23 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:
>         
>         >
>         >         I'm not sure where cybershake fits in. The problem
>         would be to
>         >         submit
>         >         dummy jobs of various walltimes to various clusters
>         and record
>         >         the
>         >         amount of time they sit in the queue.
>         >
>         > It is just a use-case. I thought, stats based on an (any)
>         real
>         > application will have more merit compared to dummy jobs.
>         >
>         
>         I don't see how it would in this case. We're trying to measure
>         queuing
>         time vs walltime in a general sense. I think it makes more
>         sense to not
>         have a specific application, which would simply be irrelevant
>         information.
> 
> Ok, we can use bionimbus cloud with Mike's catsnsleep script with
> following possible variations:
> 
> Job Completion times: sleep parameter 1s-100s
> Data : cat file size 1M-1G
> Bionimbus VMs 1-32
> 
> Number of jobs n, 1-1000

I need queuing time vs. advertised job walltime on various clusters
(with various/random degrees of utilization). That's to see whether it's
useful to have coasters at all.

The number of jobs is an orthogonal dimension (i.e. we may want to
measure the queuing time vs. #of jobs for various walltimes, but later).
The actual job duration is not relevant. The amount of data is not
relevant.

Clouds are an interesting environment, but not for this particular
problem. That's because we need to see how much it takes to acquire
resources, not how fast some job middleware is after we got hold of
those resources.



From aespinosa at cs.uchicago.edu  Mon Jul 18 01:22:23 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Mon, 18 Jul 2011 00:22:23 -0600
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310951056.7468.7.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
Message-ID: <CAAHfuQ_iYYnWxiNv4Z6Nb5kkCSaZ58HkX683krHKW5DkbrCakg@mail.gmail.com>

I did a bunch of tests last year by varying walltime (actually they
were sleep job durations) and then measure the total CPU hours
consumed over a 24 hour time window.  It was a different set of
metrics but I think we can get the data needed for the same methods.


2011/7/17 Mihael Hategan <hategan at mcs.anl.gov>:
> On Sun, 2011-07-17 at 19:53 -0500, Ketan Maheshwari wrote:
>>
>> On Sun, Jul 17, 2011 at 4:23 PM, Mihael Hategan <hategan at mcs.anl.gov>
>> wrote:
>> ? ? ? ? On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:
>>
>> ? ? ? ? >
>> ? ? ? ? > ? ? ? ? I'm not sure where cybershake fits in. The problem
>> ? ? ? ? would be to
>> ? ? ? ? > ? ? ? ? submit
>> ? ? ? ? > ? ? ? ? dummy jobs of various walltimes to various clusters
>> ? ? ? ? and record
>> ? ? ? ? > ? ? ? ? the
>> ? ? ? ? > ? ? ? ? amount of time they sit in the queue.
>> ? ? ? ? >
>> ? ? ? ? > It is just a use-case. I thought, stats based on an (any)
>> ? ? ? ? real
>> ? ? ? ? > application will have more merit compared to dummy jobs.
>> ? ? ? ? >
>>
>> ? ? ? ? I don't see how it would in this case. We're trying to measure
>> ? ? ? ? queuing
>> ? ? ? ? time vs walltime in a general sense. I think it makes more
>> ? ? ? ? sense to not
>> ? ? ? ? have a specific application, which would simply be irrelevant
>> ? ? ? ? information.
>>
>> Ok, we can use bionimbus cloud with Mike's catsnsleep script with
>> following possible variations:
>>
>> Job Completion times: sleep parameter 1s-100s
>> Data : cat file size 1M-1G
>> Bionimbus VMs 1-32
>>
>> Number of jobs n, 1-1000
>
> I need queuing time vs. advertised job walltime on various clusters
> (with various/random degrees of utilization). That's to see whether it's
> useful to have coasters at all.
>
> The number of jobs is an orthogonal dimension (i.e. we may want to
> measure the queuing time vs. #of jobs for various walltimes, but later).
> The actual job duration is not relevant. The amount of data is not
> relevant.
>
> Clouds are an interesting environment, but not for this particular
> problem. That's because we need to see how much it takes to acquire
> resources, not how fast some job middleware is after we got hold of
> those resources.
>


From hategan at mcs.anl.gov  Mon Jul 18 13:26:58 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Jul 2011 11:26:58 -0700
Subject: [Swift-devel] [Fwd: [Kraken-users] Send Kraken Jobs !!]
Message-ID: <1311013618.14422.1.camel@blabla>

I get these on occasion. This is a machine fatter than the ANL BG/P from
what I understand, and with a more down-to-earth architecture.

So I'm puzzled that this is happening.

-------- Forwarded Message --------
From: Daniel Lucio <dlucio at utk.edu>
To: kraken-users at nics.utk.edu
Subject: [Kraken-users] Send Kraken Jobs !!
Date: Mon, 18 Jul 2011 09:59:40 -0400

Kraken Users:

There are zero eligible jobs in the Kraken queues right now.  Please submit Kraken jobs. 

Daniel
NICS Team
_______________________________________________
kraken-users mailing list
kraken-users at nics.utk.edu
To unsubscribe:
https://www.nics.tennessee.edu/mailman/listinfo/kraken-users




From skenny at uchicago.edu  Mon Jul 18 13:41:25 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Mon, 18 Jul 2011 11:41:25 -0700
Subject: [Swift-devel] [Fwd: [Kraken-users] Send Kraken Jobs !!]
In-Reply-To: <1311013618.14422.1.camel@blabla>
References: <1311013618.14422.1.camel@blabla>
Message-ID: <CAE50RW1WZRXus-b6d7nQLR89ZAxeeiy8FCGRRvH_EZDVpoP-cw@mail.gmail.com>

one thing i know is that they generally don't support gateway accounts (they
told me if i was desperate they'd try to make it work :)...so, my users
can't submit jobs there under our community account..which is how we run
'cause it allows me to look at their logs and data if need be without too
much trouble.

but here's what puzzles me even more...whenever i see such notices and i
check the TG system monitor

https://portal.teragrid.org/systems-monitor

kraken appears to have lots of jobs in the queue...in fact right now it
looks like they have more than any other TG system (!)

~sk

On Mon, Jul 18, 2011 at 11:26 AM, Mihael Hategan <hategan at mcs.anl.gov>wrote:

> I get these on occasion. This is a machine fatter than the ANL BG/P from
> what I understand, and with a more down-to-earth architecture.
>
> So I'm puzzled that this is happening.
>
> -------- Forwarded Message --------
> From: Daniel Lucio <dlucio at utk.edu>
> To: kraken-users at nics.utk.edu
> Subject: [Kraken-users] Send Kraken Jobs !!
> Date: Mon, 18 Jul 2011 09:59:40 -0400
>
> Kraken Users:
>
> There are zero eligible jobs in the Kraken queues right now.  Please submit
> Kraken jobs.
>
> Daniel
> NICS Team
> _______________________________________________
> kraken-users mailing list
> kraken-users at nics.utk.edu
> To unsubscribe:
> https://www.nics.tennessee.edu/mailman/listinfo/kraken-users
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110718/affda363/attachment.html>

From hategan at mcs.anl.gov  Mon Jul 18 13:48:14 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Jul 2011 11:48:14 -0700
Subject: [Swift-devel] [Fwd: [Kraken-users] Send Kraken Jobs !!]
In-Reply-To: <CAE50RW1WZRXus-b6d7nQLR89ZAxeeiy8FCGRRvH_EZDVpoP-cw@mail.gmail.com>
References: <1311013618.14422.1.camel@blabla>
	<CAE50RW1WZRXus-b6d7nQLR89ZAxeeiy8FCGRRvH_EZDVpoP-cw@mail.gmail.com>
Message-ID: <1311014894.15080.4.camel@blabla>

On Mon, 2011-07-18 at 11:41 -0700, Sarah Kenny wrote:
> one thing i know is that they generally don't support gateway accounts
> (they told me if i was desperate they'd try to make it work :)...so,
> my users can't submit jobs there under our community account..which is
> how we run 'cause it allows me to look at their logs and data if need
> be without too much trouble. 
> but here's what puzzles me even more...whenever i see such notices and
> i check the TG system monitor
> 
> https://portal.teragrid.org/systems-monitor
> 
> kraken appears to have lots of jobs in the queue...in fact right now
> it looks like they have more than any other TG system (!)

Either one of them is lying, or the email doesn't apply any more. In any
event, it somewhat puzzles me that the situation arises in the first
place (unless, again, it's a lie).




From yadudoc1729 at gmail.com  Mon Jul 18 16:53:25 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Tue, 19 Jul 2011 03:23:25 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <1310682629.12137.1.camel@blabla>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
Message-ID: <CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>

Hi,

> Also the <variable> thing. I have a suspicion that you can use a
> function on the template to extract the contents of that tag rather than
> rendering it to a string and then parsing it again in custom code.
> So let me know if you want to fix those issues. They will have to be
> fixed sooner or later by somebody.

I've moved the code to a separate function so that the foreachstat
function in Karajan.java looks neater now. As you suggest there
might be some way to get it from the string template, but my efforts
didn't get me there.

There were a couple of multidimensional array based tests that were
failing. Multidimensional arrays with assignments of the form :
string array[ ][ ];
array = [ [ "left" , "right" ] [ "up", "down" ] ];
trace ( array[0][0], array[0][1], array[1][0], array[1][1] );

I tried to fix this, but later I tried running swift clean, with none of my
code in it, and it still failed. Should I try to fix this ?
The same for mappers, but here as Justin pointed out, the tests are
not working right in the early-revision I'm working from.


-- 
Thanks and Regards,
Yadu Nand B


From skenny at uchicago.edu  Mon Jul 18 19:53:54 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Mon, 18 Jul 2011 17:53:54 -0700
Subject: [Swift-devel] execution provider attributes
Message-ID: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>

hi all, just wanted to get a little clarification here as i'm continuing to
work on the sites file checker, chxml (
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=453)

i want to specifically add a check for the parameters that are acceptable
for a given EXECUTION provider. initially i had listed general, gt2, pbs and
coaster (general being parameters accepted by all providers including
local). but now i'm thinking that pbs is not an allowable exectution
provider. that is, i currently only want itmes that can be specified as
<execution provider="something" i know that gt2, local and coaster are legal
values for this field but can you specify pbs here? in looking deeper it
seems that the pbs parameters are generally specified with 'coasters' as the
execution provider and pbs as the jobmanager, with extra pbs params going
under 'jobAttributes'.

so, i've removed pbs from the list here
https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1 and
put the pbs parameters under jobAttributes in the coaster column. if i'm
misunderstanding and there are other legal values for execution provider
besides the 3 listed above please let me know.

thanks
~sk

On Wed, Jul 6, 2011 at 1:09 PM, Sarah Kenny <skenny at uchicago.edu> wrote:

> i've been trying to expand the job attributes list that you started,
> justin:
> https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1
>
> feel free to correct/add anything or point me to code :)
>
>
> On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:
>
>>
>> I'll be on for a call today to touch base on the latest developments...
>>
>> --
>> Justin M Wozniak
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>
>
>
> --
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
>
>


-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110718/cce12fe4/attachment.html>

From hategan at mcs.anl.gov  Mon Jul 18 20:08:35 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Jul 2011 18:08:35 -0700
Subject: [Swift-devel] execution provider attributes
In-Reply-To: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
References: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
Message-ID: <1311037715.18080.1.camel@blabla>

Pbs is a valid execution provider.

Try this if you want the complete list for your build:
cog-workflow -e 'import("sys.k"), import("task.k"), for(h,
list(from(handlers, availableHandlers(type="execution"))), echo(h))'

On Mon, 2011-07-18 at 17:53 -0700, Sarah Kenny wrote:
> hi all, just wanted to get a little clarification here as i'm
> continuing to work on the sites file checker, chxml
> (https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=453)
> 
> i want to specifically add a check for the parameters that are
> acceptable for a given EXECUTION provider. initially i had listed
> general, gt2, pbs and coaster (general being parameters accepted by
> all providers including local). but now i'm thinking that pbs is not
> an allowable exectution provider. that is, i currently only want itmes
> that can be specified as  <execution provider="something" i know that
> gt2, local and coaster are legal values for this field but can you
> specify pbs here? in looking deeper it seems that the pbs parameters
> are generally specified with 'coasters' as the execution provider and
> pbs as the jobmanager, with extra pbs params going under
> 'jobAttributes'. 
> 
> so, i've removed pbs from the list here
> https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1 and put the pbs parameters under jobAttributes in the coaster column. if i'm misunderstanding and there are other legal values for execution provider besides the 3 listed above please let me know.
> 
> thanks
> ~sk
> 
> On Wed, Jul 6, 2011 at 1:09 PM, Sarah Kenny <skenny at uchicago.edu>
> wrote:
>         i've been trying to expand the job attributes list that you
>         started, justin:
>         https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1
>         
>         feel free to correct/add anything or point me to code :)
>         
>         
>         
>         On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak
>         <wozniak at mcs.anl.gov> wrote:
>                 
>                 I'll be on for a call today to touch base on the
>                 latest developments...
>                 
>                 --
>                 Justin M Wozniak
>                 _______________________________________________
>                 Swift-devel mailing list
>                 Swift-devel at ci.uchicago.edu
>                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
>         
>         
>         
>         -- 
>         Sarah Kenny
>         Programmer
>         University of Chicago, Computation Institute
>         University of California Irvine, Dept. of Neurology
>         773-818-8300
>         
> 
> 
> 
> -- 
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Mon Jul 18 20:10:37 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 Jul 2011 18:10:37 -0700
Subject: [Swift-devel] execution provider attributes
In-Reply-To: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
References: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
Message-ID: <1311037837.18080.2.camel@blabla>

But I have to say that the attributes are things that are meant to be
flexible. Chances are you don't want to validate them much.

On Mon, 2011-07-18 at 17:53 -0700, Sarah Kenny wrote:
> hi all, just wanted to get a little clarification here as i'm
> continuing to work on the sites file checker, chxml
> (https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=453)
> 
> i want to specifically add a check for the parameters that are
> acceptable for a given EXECUTION provider. initially i had listed
> general, gt2, pbs and coaster (general being parameters accepted by
> all providers including local). but now i'm thinking that pbs is not
> an allowable exectution provider. that is, i currently only want itmes
> that can be specified as  <execution provider="something" i know that
> gt2, local and coaster are legal values for this field but can you
> specify pbs here? in looking deeper it seems that the pbs parameters
> are generally specified with 'coasters' as the execution provider and
> pbs as the jobmanager, with extra pbs params going under
> 'jobAttributes'. 
> 
> so, i've removed pbs from the list here
> https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1 and put the pbs parameters under jobAttributes in the coaster column. if i'm misunderstanding and there are other legal values for execution provider besides the 3 listed above please let me know.
> 
> thanks
> ~sk
> 
> On Wed, Jul 6, 2011 at 1:09 PM, Sarah Kenny <skenny at uchicago.edu>
> wrote:
>         i've been trying to expand the job attributes list that you
>         started, justin:
>         https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1
>         
>         feel free to correct/add anything or point me to code :)
>         
>         
>         
>         On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak
>         <wozniak at mcs.anl.gov> wrote:
>                 
>                 I'll be on for a call today to touch base on the
>                 latest developments...
>                 
>                 --
>                 Justin M Wozniak
>                 _______________________________________________
>                 Swift-devel mailing list
>                 Swift-devel at ci.uchicago.edu
>                 https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>         
>         
>         
>         
>         -- 
>         Sarah Kenny
>         Programmer
>         University of Chicago, Computation Institute
>         University of California Irvine, Dept. of Neurology
>         773-818-8300
>         
> 
> 
> 
> -- 
> Sarah Kenny
> Programmer
> University of Chicago, Computation Institute
> University of California Irvine, Dept. of Neurology
> 773-818-8300
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Tue Jul 19 13:56:05 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 19 Jul 2011 11:56:05 -0700
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
	<CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
Message-ID: <1311101765.20818.1.camel@blabla>

On Tue, 2011-07-19 at 03:23 +0530, Yadu Nand wrote:
> Hi,
> 
> > Also the <variable> thing. I have a suspicion that you can use a
> > function on the template to extract the contents of that tag rather than
> > rendering it to a string and then parsing it again in custom code.
> > So let me know if you want to fix those issues. They will have to be
> > fixed sooner or later by somebody.
> 
> I've moved the code to a separate function so that the foreachstat
> function in Karajan.java looks neater now. As you suggest there
> might be some way to get it from the string template, but my efforts
> didn't get me there.
> 
> There were a couple of multidimensional array based tests that were
> failing. Multidimensional arrays with assignments of the form :
> string array[ ][ ];
> array = [ [ "left" , "right" ] [ "up", "down" ] ];
> trace ( array[0][0], array[0][1], array[1][0], array[1][1] );
> 
> I tried to fix this, but later I tried running swift clean, with none of my
> code in it, and it still failed. Should I try to fix this ?
> The same for mappers, but here as Justin pointed out, the tests are
> not working right in the early-revision I'm working from.
> 

Right. It may be that other things are broken in the version of swift
you're working against. If you think the code is ok, send me a patch and
I'll merge it to my local checkout and see if I can fix the problems.




From yadudoc1729 at gmail.com  Wed Jul 20 00:42:52 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Wed, 20 Jul 2011 11:12:52 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <1311101765.20818.1.camel@blabla>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
	<CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
	<1311101765.20818.1.camel@blabla>
Message-ID: <CANa904nGOhCc6y7_V4yTbLJ4u1gSJP8z2NN1XSFM-r0Bsh7Hbg@mail.gmail.com>

> Right. It may be that other things are broken in the version of swift
> you're working against. If you think the code is ok, send me a patch and
> I'll merge it to my local checkout and see if I can fix the problems.

Great!
Please find the patch attached. I see that there are a lot of changes which
are actually due to some extra white-spaces (possibly), which will make
reading the actual diff a bit difficult.

-- 
Thanks and Regards,
Yadu Nand B


From yadudoc1729 at gmail.com  Wed Jul 20 01:09:15 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Wed, 20 Jul 2011 11:39:15 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904nGOhCc6y7_V4yTbLJ4u1gSJP8z2NN1XSFM-r0Bsh7Hbg@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
	<CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
	<1311101765.20818.1.camel@blabla>
	<CANa904nGOhCc6y7_V4yTbLJ4u1gSJP8z2NN1XSFM-r0Bsh7Hbg@mail.gmail.com>
Message-ID: <CANa904nBj1w6DZo-0x+E8a=AdDNuSuSvru2q8rdtfi=qScLK-g@mail.gmail.com>

> Great!
> Please find the patch attached. I see that there are a lot of changes which
> are actually due to some extra white-spaces (possibly), which will make
> reading the actual diff a bit difficult.
My bad, forgot the patch again. :(

-- 
Thanks and Regards,
Yadu Nand B
-------------- next part --------------
A non-text attachment was scrubbed...
Name: seventh.patch
Type: text/x-patch
Size: 143475 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110720/467302fc/attachment.bin>

From skenny at uchicago.edu  Wed Jul 20 14:50:05 2011
From: skenny at uchicago.edu (Sarah Kenny)
Date: Wed, 20 Jul 2011 12:50:05 -0700
Subject: [Swift-devel] execution provider attributes
In-Reply-To: <1311037837.18080.2.camel@blabla>
References: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
	<1311037837.18080.2.camel@blabla>
Message-ID: <CAE50RW1jUjz9hTF5j8oKvjE6389xecgwP5PqGvriyBy42Y9WAA@mail.gmail.com>

sooo, we don't want to validate attributes for execution providers (bcs they
will change a lot)? or we just don't want to validate things specified with
the 'jobAttributes' key? if it's the former i'm not *quite* sure what to do
with this bug, since it says

"Swift should warn the user (or give a fatal error) if parameters specified
in the sites file dont apply to the selected provider(s)."

if the attribute list is meant to be dynamic, is it possible to extract the
parameters of an execution provider for a given version by *somehow*
searching thru the user's swift installation?

sorry if i'm missing something obvious here...

~sk

On Mon, Jul 18, 2011 at 6:10 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> But I have to say that the attributes are things that are meant to be
> flexible. Chances are you don't want to validate them much.
>
> On Mon, 2011-07-18 at 17:53 -0700, Sarah Kenny wrote:
> > hi all, just wanted to get a little clarification here as i'm
> > continuing to work on the sites file checker, chxml
> > (https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=453)
> >
> > i want to specifically add a check for the parameters that are
> > acceptable for a given EXECUTION provider. initially i had listed
> > general, gt2, pbs and coaster (general being parameters accepted by
> > all providers including local). but now i'm thinking that pbs is not
> > an allowable exectution provider. that is, i currently only want itmes
> > that can be specified as  <execution provider="something" i know that
> > gt2, local and coaster are legal values for this field but can you
> > specify pbs here? in looking deeper it seems that the pbs parameters
> > are generally specified with 'coasters' as the execution provider and
> > pbs as the jobmanager, with extra pbs params going under
> > 'jobAttributes'.
> >
> > so, i've removed pbs from the list here
> > https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1and put the pbs parameters under jobAttributes in the coaster column. if i'm
> misunderstanding and there are other legal values for execution provider
> besides the 3 listed above please let me know.
> >
> > thanks
> > ~sk
> >
> > On Wed, Jul 6, 2011 at 1:09 PM, Sarah Kenny <skenny at uchicago.edu>
> > wrote:
> >         i've been trying to expand the job attributes list that you
> >         started, justin:
> >
> https://sites.google.com/site/swiftdevel/internals/job-attributes?pli=1
> >
> >         feel free to correct/add anything or point me to code :)
> >
> >
> >
> >         On Wed, Jul 6, 2011 at 12:36 PM, Justin M Wozniak
> >         <wozniak at mcs.anl.gov> wrote:
> >
> >                 I'll be on for a call today to touch base on the
> >                 latest developments...
> >
> >                 --
> >                 Justin M Wozniak
> >                 _______________________________________________
> >                 Swift-devel mailing list
> >                 Swift-devel at ci.uchicago.edu
> >
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> >
> >
> >         --
> >         Sarah Kenny
> >         Programmer
> >         University of Chicago, Computation Institute
> >         University of California Irvine, Dept. of Neurology
> >         773-818-8300
> >
> >
> >
> >
> > --
> > Sarah Kenny
> > Programmer
> > University of Chicago, Computation Institute
> > University of California Irvine, Dept. of Neurology
> > 773-818-8300
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>


-- 
Sarah Kenny
Programmer
University of Chicago, Computation Institute
University of California Irvine, Dept. of Neurology
773-818-8300
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110720/70a9f8c0/attachment.html>

From hategan at mcs.anl.gov  Wed Jul 20 15:16:22 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 20 Jul 2011 13:16:22 -0700
Subject: [Swift-devel] execution provider attributes
In-Reply-To: <CAE50RW1jUjz9hTF5j8oKvjE6389xecgwP5PqGvriyBy42Y9WAA@mail.gmail.com>
References: <CAE50RW0m_xV2YiysSC+0WW78bJg8+dqvGNXCWMre0YVBXPjHTw@mail.gmail.com>
	<1311037837.18080.2.camel@blabla>
	<CAE50RW1jUjz9hTF5j8oKvjE6389xecgwP5PqGvriyBy42Y9WAA@mail.gmail.com>
Message-ID: <1311192982.27601.5.camel@blabla>

On Wed, 2011-07-20 at 12:50 -0700, Sarah Kenny wrote:
> sooo, we don't want to validate attributes for execution providers
> (bcs they will change a lot)? or we just don't want to validate things
> specified with the 'jobAttributes' key? if it's the former i'm not
> *quite* sure what to do with this bug, since it says
> 
> "Swift should warn the user (or give a fatal error) if parameters
> specified in the sites file dont apply to the selected provider(s)."

Hmm. Well, maybe the providers should complain about that and also
provide a list of supported attributes.


> if the attribute list is meant to be dynamic, is it possible to
> extract the parameters of an execution provider for a given version by
> *somehow* searching thru the user's swift installation?

What you could do is file a bug report for such introspective features
for the providers, and I'll handle that and then you can take it from
there.




From wozniak at mcs.anl.gov  Wed Jul 20 15:33:31 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 20 Jul 2011 15:33:31 -0500 (CDT)
Subject: [Swift-devel] No call today - bugzilla
Message-ID: <alpine.DEB.2.00.1107201532400.1980@wozniak-desktop.mcs.anl.gov>


Let's skip the call for today but continue going through bugzilla and 
narrowing down the scope for 0.93 ...

-- 
Justin M Wozniak


From ketancmaheshwari at gmail.com  Wed Jul 20 16:41:13 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Wed, 20 Jul 2011 16:41:13 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <1310951056.7468.7.camel@blabla>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
Message-ID: <CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>

>
> I need queuing time vs. advertised job walltime on various clusters
> (with various/random degrees of utilization). That's to see whether it's
> useful to have coasters at all.
>
> The number of jobs is an orthogonal dimension (i.e. we may want to
> measure the queuing time vs. #of jobs for various walltimes, but later).
> The actual job duration is not relevant. The amount of data is not
> relevant.
>
> Clouds are an interesting environment, but not for this particular
> problem. That's because we need to see how much it takes to acquire
> resources, not how fast some job middleware is after we got hold of
> those resources.
>
>
Do you have any specific environments in mind for these experiments? For the
requirement of various/random degrees of utilization, we can use MCS local
cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger.

Regards,
-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110720/d2ed2bc3/attachment.html>

From hategan at mcs.anl.gov  Wed Jul 20 17:03:24 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 20 Jul 2011 15:03:24 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
	<CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
Message-ID: <1311199404.3530.8.camel@blabla>

On Wed, 2011-07-20 at 16:41 -0500, Ketan Maheshwari wrote:

>         
>         The number of jobs is an orthogonal dimension (i.e. we may
>         want to
>         measure the queuing time vs. #of jobs for various walltimes,
>         but later).
>         The actual job duration is not relevant. The amount of data is
>         not
>         relevant.
>         
>         Clouds are an interesting environment, but not for this
>         particular
>         problem. That's because we need to see how much it takes to
>         acquire
>         resources, not how fast some job middleware is after we got
>         hold of
>         those resources.
>         
> 
> Do you have any specific environments in mind for these experiments?
> For the requirement of various/random degrees of utilization, we can
> use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and
> Ranger. 

Not really besides them being clusters with multiple users.

When there are fewer resources than demand, some economics needs to
happen. This is mostly visible in queuing times (a user has nothing else
to pay with than time - assuming a working environment). There are
multiple things that influence that cost, but I only want to know how
requested walltimes affect it. Since we can't directly isolate that from
other things, he have to employ statistics. The more random the
clusters/times/<any other variables that could affect queueing times>,
the better we can infer some general formula for things we're interested
in (because whatever is common to all these environments is likely to be
the effect of the parameter we're trying to play with - walltime).




From foster at anl.gov  Thu Jul 21 09:18:01 2011
From: foster at anl.gov (Ian Foster)
Date: Thu, 21 Jul 2011 09:18:01 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
	<CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
Message-ID: <F94E4491-3CC8-4911-BB78-8A0A278B241F@anl.gov>

I think that you may find it useful to look at the queue time predictor that Rich Wolski developed. QBETS I think is the name?

Ian.

On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote:

> 
> 
> I need queuing time vs. advertised job walltime on various clusters
> (with various/random degrees of utilization). That's to see whether it's
> useful to have coasters at all.
> 
> The number of jobs is an orthogonal dimension (i.e. we may want to
> measure the queuing time vs. #of jobs for various walltimes, but later).
> The actual job duration is not relevant. The amount of data is not
> relevant.
> 
> Clouds are an interesting environment, but not for this particular
> problem. That's because we need to see how much it takes to acquire
> resources, not how fast some job middleware is after we got hold of
> those resources.
> 
> 
> Do you have any specific environments in mind for these experiments? For the requirement of various/random degrees of utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger. 
> 
> Regards,
> -- 
> Ketan
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110721/475667d0/attachment.html>

From foster at anl.gov  Thu Jul 21 09:19:22 2011
From: foster at anl.gov (Ian Foster)
Date: Thu, 21 Jul 2011 09:19:22 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <F94E4491-3CC8-4911-BB78-8A0A278B241F@anl.gov>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
	<CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
	<F94E4491-3CC8-4911-BB78-8A0A278B241F@anl.gov>
Message-ID: <E864053D-AD90-4156-B057-B5CD5D006E3C@anl.gov>

E.g., I used it here: http://ianfoster.typepad.com/blog/2009/08/whats-fastera-supercomputer-or-ec2.html


On Jul 21, 2011, at 9:18 AM, Ian Foster wrote:

> I think that you may find it useful to look at the queue time predictor that Rich Wolski developed. QBETS I think is the name?
> 
> Ian.
> 
> On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote:
> 
>> 
>> 
>> I need queuing time vs. advertised job walltime on various clusters
>> (with various/random degrees of utilization). That's to see whether it's
>> useful to have coasters at all.
>> 
>> The number of jobs is an orthogonal dimension (i.e. we may want to
>> measure the queuing time vs. #of jobs for various walltimes, but later).
>> The actual job duration is not relevant. The amount of data is not
>> relevant.
>> 
>> Clouds are an interesting environment, but not for this particular
>> problem. That's because we need to see how much it takes to acquire
>> resources, not how fast some job middleware is after we got hold of
>> those resources.
>> 
>> 
>> Do you have any specific environments in mind for these experiments? For the requirement of various/random degrees of utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger. 
>> 
>> Regards,
>> -- 
>> Ketan
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110721/25fa5b25/attachment.html>

From wilde at mcs.anl.gov  Thu Jul 21 11:12:19 2011
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 21 Jul 2011 11:12:19 -0500 (CDT)
Subject: [Swift-devel] pilot job paper
In-Reply-To: <E864053D-AD90-4156-B057-B5CD5D006E3C@anl.gov>
Message-ID: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>

Ian, QBETS sounds very useful; I met the developer at last year's SDCI meeting, and thought we could readily use QBETS predictions in Swift scheduling. Mihael, what would be a general interface from picking up site queue predictions from such tools with some kind of loose coupling? 


Also, in the Condor-G-based coaster factory approach we tested in ExTENCI, we could plug QBETS info into its factory script to bias the volume of pilot jobs we are sending to each site. Then the mod Mihael made recently in coaster scheduling will automatically keep all running coaster slots filled with active jobs. 



- Mike 

----- Original Message -----



E.g., I used it here: http://ianfoster.typepad.com/blog/2009/08/whats-fastera-supercomputer-or-ec2.html 




On Jul 21, 2011, at 9:18 AM, Ian Foster wrote: 



I think that you may find it useful to look at the queue time predictor that Rich Wolski developed. QBETS I think is the name? 


Ian. 



On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote: 









I need queuing time vs. advertised job walltime on various clusters 
(with various/random degrees of utilization). That's to see whether it's 
useful to have coasters at all. 

The number of jobs is an orthogonal dimension (i.e. we may want to 
measure the queuing time vs. #of jobs for various walltimes, but later). 
The actual job duration is not relevant. The amount of data is not 
relevant. 

Clouds are an interesting environment, but not for this particular 
problem. That's because we need to see how much it takes to acquire 
resources, not how fast some job middleware is after we got hold of 
those resources. 


Do you have any specific environments in mind for these experiments? For the requirement of various/random degrees of utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger. 

Regards, 
-- 
Ketan 


_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 

_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 


_______________________________________________ 
Swift-devel mailing list 
Swift-devel at ci.uchicago.edu 
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel 



-- 
Michael Wilde 
Computation Institute, University of Chicago 
Mathematics and Computer Science Division 
Argonne National Laboratory 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110721/c5d63d3f/attachment.html>

From foster at anl.gov  Thu Jul 21 11:41:33 2011
From: foster at anl.gov (Ian Foster)
Date: Thu, 21 Jul 2011 11:41:33 -0500
Subject: [Swift-devel] pilot job paper
In-Reply-To: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
References: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
Message-ID: <3E842E78-C617-4375-8E5C-7368406AF1D7@anl.gov>

I made the suggestion in the context of the discussion of data sources about historical queue times for a paper. But I can imagine it could be useful for scheduling too.


On Jul 21, 2011, at 11:12 AM, Michael Wilde wrote:

> Ian, QBETS sounds very useful; I met the developer at last year's SDCI meeting, and thought we could readily use QBETS predictions in Swift scheduling. Mihael, what would be a general interface from picking up site queue predictions from such tools with some kind of loose coupling?
> 
> Also, in the Condor-G-based coaster factory approach we tested in ExTENCI, we could plug QBETS info into its factory script to bias the volume of pilot jobs we are sending to each site. Then the mod Mihael made recently in coaster scheduling will automatically keep all running coaster slots filled with active jobs.
> 
> - Mike
> 
> E.g., I used it here: http://ianfoster.typepad.com/blog/2009/08/whats-fastera-supercomputer-or-ec2.html
> 
> 
> On Jul 21, 2011, at 9:18 AM, Ian Foster wrote:
> 
> I think that you may find it useful to look at the queue time predictor that Rich Wolski developed. QBETS I think is the name?
> 
> Ian.
> 
> On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote:
> 
> 
> 
> I need queuing time vs. advertised job walltime on various clusters
> (with various/random degrees of utilization). That's to see whether it's
> useful to have coasters at all.
> 
> The number of jobs is an orthogonal dimension (i.e. we may want to
> measure the queuing time vs. #of jobs for various walltimes, but later).
> The actual job duration is not relevant. The amount of data is not
> relevant.
> 
> Clouds are an interesting environment, but not for this particular
> problem. That's because we need to see how much it takes to acquire
> resources, not how fast some job middleware is after we got hold of
> those resources.
> 
> 
> Do you have any specific environments in mind for these experiments? For the requirement of various/random degrees of utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger. 
> 
> Regards,
> -- 
> Ketan
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110721/09b4f9d8/attachment.html>

From dsk at ci.uchicago.edu  Thu Jul 21 11:52:00 2011
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Thu, 21 Jul 2011 10:52:00 -0600
Subject: [Swift-devel] pilot job paper
In-Reply-To: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
References: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
Message-ID: <CE505E9D-2F11-48F6-9D75-CF8E447B550C@ci.uchicago.edu>


FYI. This is no longer supported. TG is now using Karnak, from Warren Smith, for this functionality. 


On Jul 21, 2011, at 10:12, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Ian, QBETS sounds very useful; I met the developer at last year's SDCI meeting, and thought we could readily use QBETS predictions in Swift scheduling. Mihael, what would be a general interface from picking up site queue predictions from such tools with some kind of loose coupling?
> 
> Also, in the Condor-G-based coaster factory approach we tested in ExTENCI, we could plug QBETS info into its factory script to bias the volume of pilot jobs we are sending to each site. Then the mod Mihael made recently in coaster scheduling will automatically keep all running coaster slots filled with active jobs.
> 
> - Mike
> 
> E.g., I used it here: http://ianfoster.typepad.com/blog/2009/08/whats-fastera-supercomputer-or-ec2.html
> 
> 
> On Jul 21, 2011, at 9:18 AM, Ian Foster wrote:
> 
> I think that you may find it useful to look at the queue time predictor that Rich Wolski developed. QBETS I think is the name?
> 
> Ian.
> 
> On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote:
> 
> 
> 
> I need queuing time vs. advertised job walltime on various clusters
> (with various/random degrees of utilization). That's to see whether it's
> useful to have coasters at all.
> 
> The number of jobs is an orthogonal dimension (i.e. we may want to
> measure the queuing time vs. #of jobs for various walltimes, but later).
> The actual job duration is not relevant. The amount of data is not
> relevant.
> 
> Clouds are an interesting environment, but not for this particular
> problem. That's because we need to see how much it takes to acquire
> resources, not how fast some job middleware is after we got hold of
> those resources.
> 
> 
> Do you have any specific environments in mind for these experiments? For the requirement of various/random degrees of utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit machines), Beagle, and Ranger. 
> 
> Regards,
> -- 
> Ketan
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110721/171604b5/attachment.html>

From hategan at mcs.anl.gov  Thu Jul 21 12:36:10 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 21 Jul 2011 10:36:10 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <E864053D-AD90-4156-B057-B5CD5D006E3C@anl.gov>
References: <351209163.101339.1310928744910.JavaMail.root@zimbra.anl.gov>
	<1310929185.32625.1.camel@blabla> <1310929652.32736.1.camel@blabla>
	<CAMUuvipWM84BLfOWZMcviTDXtBj-g_RKqDA6UewmTqdN90O7YA@mail.gmail.com>
	<1310937268.1163.11.camel@blabla>
	<CAMUuvipGowtk8DFZXm4iQJO+kdbafzX-_A2zHCgTSrBjzYFW_A@mail.gmail.com>
	<1310937813.1346.4.camel@blabla>
	<CAMUuvir_mYppW5mrmUfSDYWGc0ckD4v61F5pUzAsPKFaQr18eQ@mail.gmail.com>
	<1310951056.7468.7.camel@blabla>
	<CAMUuviqFHq_fbX0ekag4Oh2=+pzqnwueVFh6NAgtc-Fv3nH3iQ@mail.gmail.com>
	<F94E4491-3CC8-4911-BB78-8A0A278B241F@anl.gov>
	<E864053D-AD90-4156-B057-B5CD5D006E3C@anl.gov>
Message-ID: <1311269770.6744.7.camel@blabla>

Thanks.

I think I tried previously to find the raw data they used, but couldn't
get much out of the paper in that direction. And that seems to be
essential for the numbers I want. One could use QBETS (if it was exposed
as a service) to get the needed data instead of running the actual test,
but I think that would be a weaker argument.

Ultimately one can fall back to the argument that there is some constant
cost in all cases for queuing systems, a cost which is relevant for jobs
of the order of 10s, but it would be more interesting to show that this
is a good strategy for a wider range of jobs.

Mihael

On Thu, 2011-07-21 at 09:19 -0500, Ian Foster wrote:
> E.g., I used it
> here: http://ianfoster.typepad.com/blog/2009/08/whats-fastera-supercomputer-or-ec2.html
> 
> 
> 
> On Jul 21, 2011, at 9:18 AM, Ian Foster wrote:
> 
> > I think that you may find it useful to look at the queue time
> > predictor that Rich Wolski developed. QBETS I think is the name?
> > 
> > 
> > Ian.
> > 
> > On Jul 20, 2011, at 4:41 PM, Ketan Maheshwari wrote:
> > 
> > > 
> > >         
> > >         
> > >         I need queuing time vs. advertised job walltime on various
> > >         clusters
> > >         (with various/random degrees of utilization). That's to
> > >         see whether it's
> > >         useful to have coasters at all.
> > >         
> > >         The number of jobs is an orthogonal dimension (i.e. we may
> > >         want to
> > >         measure the queuing time vs. #of jobs for various
> > >         walltimes, but later).
> > >         The actual job duration is not relevant. The amount of
> > >         data is not
> > >         relevant.
> > >         
> > >         Clouds are an interesting environment, but not for this
> > >         particular
> > >         problem. That's because we need to see how much it takes
> > >         to acquire
> > >         resources, not how fast some job middleware is after we
> > >         got hold of
> > >         those resources.
> > >         
> > > 
> > > Do you have any specific environments in mind for these
> > > experiments? For the requirement of various/random degrees of
> > > utilization, we can use MCS local cluster (10 x 64bit + 3 x 32 bit
> > > machines), Beagle, and Ranger. 
> > > 
> > > Regards,
> > > -- 
> > > Ketan
> > > 
> > > 
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Thu Jul 21 12:49:14 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 21 Jul 2011 10:49:14 -0700
Subject: [Swift-devel] pilot job paper
In-Reply-To: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
References: <226742456.113856.1311264739599.JavaMail.root@zimbra.anl.gov>
Message-ID: <1311270554.6744.19.camel@blabla>

On Thu, 2011-07-21 at 11:12 -0500, Michael Wilde wrote:
> Ian, QBETS sounds very useful; I met the developer at last year's SDCI
> meeting, and thought we could readily use QBETS predictions in Swift
> scheduling. Mihael, what would be a general interface from picking up
> site queue predictions from such tools with some kind of loose
> coupling?

Populate initial scores and job throttles based on QBETS data instead of
starting from scratch in every run.
> 
> 
> Also, in the Condor-G-based coaster factory approach we tested in
> ExTENCI, we could plug QBETS info into its factory script to bias the
> volume of pilot jobs we are sending to each site. Then the mod Mihael
> made recently in coaster scheduling will automatically keep all
> running coaster slots filled with active jobs.

Right. But let's focus on the paper?






From hategan at mcs.anl.gov  Thu Jul 21 12:53:26 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 21 Jul 2011 10:53:26 -0700
Subject: [Swift-devel] MCP
Message-ID: <1311270806.6744.21.camel@blabla>

Heh: replication: https://www.teragrid.org/web/user-support/mcp




From hategan at mcs.anl.gov  Fri Jul 22 14:19:50 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 22 Jul 2011 12:19:50 -0700
Subject: [Swift-devel] typed keys
Message-ID: <1311362390.23447.11.camel@blabla>

While integrating Yadu's patch, I came across the following issue with
appends:

When you do the auto-increment thing, the key type is a string (due to
the determinism constraints). When you declare an array with implicit
type, we decided that's an int. So, while the run-time will happily
append to an array[], that really breaks the typing when you do a
foreach v, k in array {}. I think this is why Yadu was suggesting a hash
- to get the key to be an int eventually.

There are a few solutions I can think of:
1. Have a universal type, which is what is used when saying array[].
That would allow both array[0] = value, array["key"] = value, and array
<< value. Also allow auto-increment on array[string]. This will keep
compatibility with existing scripts (i.e. allowing int keys when there
is no explicit key type in the declaration). This would require us to be
clear about when and how the universal/top type can be used.
2. Have an auto-increment key type and only allow << on those. I believe
that Ben mentioned something like this.
3. Require that arrays supporting << be array[string] and keep array[]
as array[int].

Thoughts?



From yadudoc1729 at gmail.com  Sat Jul 23 12:37:12 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Sat, 23 Jul 2011 23:07:12 +0530
Subject: [Swift-devel] typed keys
In-Reply-To: <1311362390.23447.11.camel@blabla>
References: <1311362390.23447.11.camel@blabla>
Message-ID: <CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>

> There are a few solutions I can think of:
> 1. Have a universal type, which is what is used when saying array[].
> That would allow both array[0] = value, array["key"] = value, and array
> << value. Also allow auto-increment on array[string]. This will keep
> compatibility with existing scripts (i.e. allowing int keys when there
> is no explicit key type in the declaration). This would require us to be
> clear about when and how the universal/top type can be used.
Are you suggesting something like an object type ? But wouldn't that
reduce the strongly-typed 'ness of the language ?

> 2. Have an auto-increment key type and only allow << on those. I believe
> that Ben mentioned something like this.
Why do we need a separate type ? Or, are you saying the user would say
array[int]["auto-inc"] and we'd internally use strings ?

> 3. Require that arrays supporting << be array[string] and keep array[]
> as array[int].
This is probably the simplest way to go about it. I could perhaps add
some extra checks to ensure that the subscript has been declared as
string.

This is probably a language decision which I shouldn't influence
because I tend to lean towards ease of implementation :)

-- 
Thanks and Regards,
Yadu Nand B


From hategan at mcs.anl.gov  Sat Jul 23 14:30:44 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 23 Jul 2011 12:30:44 -0700
Subject: [Swift-devel] typed keys
In-Reply-To: <CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>
References: <1311362390.23447.11.camel@blabla>
	<CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>
Message-ID: <1311449444.4930.16.camel@blabla>

On Sat, 2011-07-23 at 23:07 +0530, Yadu Nand wrote:
> > There are a few solutions I can think of:
> > 1. Have a universal type, which is what is used when saying array[].
> > That would allow both array[0] = value, array["key"] = value, and array
> > << value. Also allow auto-increment on array[string]. This will keep
> > compatibility with existing scripts (i.e. allowing int keys when there
> > is no explicit key type in the declaration). This would require us to be
> > clear about when and how the universal/top type can be used.
> Are you suggesting something like an object type ? But wouldn't that
> reduce the strongly-typed 'ness of the language ?

Yes. Something like Object in Java. And not necessarily: "Luca
Cardelli's article Typeful Programming describes strong typing simply as
the absence of unchecked run-time type
errors." (http://en.wikipedia.org/wiki/Strong_typing#Interpretation)

Though I don't think the language allows run-time type errors to happen.

The only issue is that we need predictability of the ordering of array
elements when passed to an app. For example "echo array" should always
produce the list of array elements according to a well defined ordering
on the keys. This is easier to do when the keys are of the same type,
but I suppose it can also be done with mixed types if there is an
ordering on the types themselves.

> 
> > 2. Have an auto-increment key type and only allow << on those. I believe
> > that Ben mentioned something like this.
> Why do we need a separate type ? Or, are you saying the user would say
> array[int]["auto-inc"] and we'd internally use strings ?

Something like that. I went with strings for now, but I think this is
the cleanest solution in the end. Though I'm a bit fuzzy. This would
also mean that we wouldn't allow a["somekey"] = v, and the only way to
use such an array would be with keys of some black-box value of type
"autokey". Which may be an advantage, since allowing manual keying of
auto arrays may generate involuntary collisions (if somebody decides to
do use a key that's the same as an auto-generated key).

Ben, Ian?

> 
> > 3. Require that arrays supporting << be array[string] and keep array[]
> > as array[int].
> This is probably the simplest way to go about it. I could perhaps add
> some extra checks to ensure that the subscript has been declared as
> string.

I added the checks.

> 
> This is probably a language decision which I shouldn't influence
> because I tend to lean towards ease of implementation :)
> 

Well, maybe. The way I see it is that sacrifices in the ease of
implementation may be worth making if they make the user's life better.
After all, in theory, the implementation is re-usable, while users
writing programs is something that happens over and over. In other
words, if you use a tool a lot, you might as well make it a good tool.



From benc at hawaga.org.uk  Sun Jul 24 03:53:56 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 24 Jul 2011 10:53:56 +0200
Subject: [Swift-devel] typed keys
In-Reply-To: <1311449444.4930.16.camel@blabla>
References: <1311362390.23447.11.camel@blabla>
	<CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>
	<1311449444.4930.16.camel@blabla>
Message-ID: <FC274AF4-A826-4E04-A880-55D982CCF6DF@hawaga.org.uk>

OK.

First, there's a fourth alternative, that that [] means that the type inside that [] should be inferred (eg using something like hindley-millner] - from a user perspective that might look a bit like hategan's option 1: (have a universal type) but its keeps the array indices a single type: you can declare:

int a[]; a["f"]=10; a["q"]=20; or int a[]; a[4]=10; a[5]=20; but not int a[]; a["f"]=10; a[5]=20;

Type inference can introduce awkward-for-the-user error messages, though (which is what irked me when Milena prototyped it a few years ago for her summer-of-code project): The three statements:
 int a[]; a["f"]=10; a[5]=20;
are wrong, but in which of the three statements is there an error?

Maybe in this restricted case its ok, though?

I dislike the idea of introducing a type hierarchy without a lot more thinking of what it affects. For example, I think it can then introduce runtime type errors: 5+"monkey" (for example) or inability to add two numbers (5+5 = TYPE ERROR! because those 5s happen to be Objects not numbers)

If I had to choose between having a String and having a MagicScopeID type to represent the auto generated ID, I think I'd probabyl prefer the magicscopeid type. But I'm not massively convinced. Because although it produces a nice type separation, it doesn't stop you acquiring magicscopeid values. Although you can't construct them yourself, you can still extract them inside a foreach loop and do bad things: (a concern hategan raised):

 int a[magic]; int b[magic]; int c[magic];
populate(a); populate(b);
foreach v,k in a {
  foreach u,l in b {
    c[k] = "hello";
  }
}
will attempt to create duplicate entries in c, which is not permitted - so its not actually introducing much protection.
On the other hand, using a string ties it into the present implementation in a way that I think need not happen, and if you require explicit type declaration rather than [], then why does [string] make any more sense than [magicauto] ? (perhaps compare to how pointers are stored as ints; but you write foo *a; rather than  "int a; /* which will point to a foo */")

I think explicitly declaring the type of array indices is probably ok. You declare the type of pretty much everything else when you write the program, so why not this? The closest example in Java I can think of is maps that use Java generics:

    HashMap<Integer,String> map = new HashMap<Integer,String>();

where you declare both the key type and the value type.

Ben



From hategan at mcs.anl.gov  Sun Jul 24 05:09:00 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 24 Jul 2011 03:09:00 -0700
Subject: [Swift-devel] typed keys
In-Reply-To: <FC274AF4-A826-4E04-A880-55D982CCF6DF@hawaga.org.uk>
References: <1311362390.23447.11.camel@blabla>
	<CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>
	<1311449444.4930.16.camel@blabla>
	<FC274AF4-A826-4E04-A880-55D982CCF6DF@hawaga.org.uk>
Message-ID: <1311502140.10830.20.camel@blabla>

On Sun, 2011-07-24 at 10:53 +0200, Ben Clifford wrote:
> OK.

This is good...
> 
> First, there's a fourth alternative, that that [] means that the type
> inside that [] should be inferred (eg using something like
> hindley-millner] - from a user perspective that might look a bit like
> hategan's option 1: (have a universal type) but its keeps the array
> indices a single type: you can declare:
> 
> int a[]; a["f"]=10; a["q"]=20; or int a[]; a[4]=10; a[5]=20; but not int a[]; a["f"]=10; a[5]=20;

Yes. That's a possibility. Though I was trying to avoid it at this
point.

> 
> Type inference can introduce awkward-for-the-user error messages,
> though (which is what irked me when Milena prototyped it a few years
> ago for her summer-of-code project): The three statements:
>  int a[]; a["f"]=10; a[5]=20;
> are wrong, but in which of the three statements is there an error?

None in particular. Or rather said all. It's not like a type error (or
any kind of error) is necessarily restricted to a single statement in
the code. An error message could print out both lines as equally
probable to have contributed to the problem. So I don't think this by
itself is a big problem.

> 
> Maybe in this restricted case its ok, though?
> 
> I dislike the idea of introducing a type hierarchy without a lot more
> thinking of what it affects. For example, I think it can then
> introduce runtime type errors: 5+"monkey" (for example) or inability
> to add two numbers (5+5 = TYPE ERROR! because those 5s happen to be
> Objects not numbers)

You would not be able to use these things in any other context than the
places that accept the any type. Such as trace() and whatever else is
there that fits, but not "+".

> 
> If I had to choose between having a String and having a MagicScopeID
> type to represent the auto generated ID, I think I'd probabyl prefer
> the magicscopeid type. But I'm not massively convinced. Because
> although it produces a nice type separation, it doesn't stop you
> acquiring magicscopeid values. Although you can't construct them
> yourself, you can still extract them inside a foreach loop and do bad
> things: (a concern hategan raised):
> 
>  int a[magic]; int b[magic]; int c[magic];
> populate(a); populate(b);
> foreach v,k in a {
>   foreach u,l in b {
>     c[k] = "hello";
>   }
> }
> will attempt to create duplicate entries in c, which is not permitted - so its not actually introducing much protection.

True. But you will get a runtime error. It is not unlike the same
scenario using a[int], b[int]. So it is not introducing
<b>additional</b> problems.

> On the other hand, using a string ties it into the present
> implementation in a way that I think need not happen, and if you
> require explicit type declaration rather than [], then why does
> [string] make any more sense than [magicauto] ? (perhaps compare to
> how pointers are stored as ints; but you write foo *a; rather than
> "int a; /* which will point to a foo */")

Right. Which is why I actually went with auto after sending the email. I
just didn't have a nice name for it, but I suppose "auto" will have to
do. I also reasoned that this would be used sparsely, so it's more
important to make sure everything else makes sense.

> 
> I think explicitly declaring the type of array indices is probably ok.
> You declare the type of pretty much everything else when you write the
> program, so why not this? The closest example in Java I can think of
> is maps that use Java generics:
> 
>     HashMap<Integer,String> map = new HashMap<Integer,String>();
> 
> where you declare both the key type and the value type.

Well, ML would disagree. I think Swift is/can be somewhere in the
middle. At least we have type inference for "foreach". Also, I don't
think some simple type inference is that bad in terms of clarity. It
gets bad when you start inferring based on expressions/calls rather than
just assignment, in particular when you have overloading. I think one
can have some simple type inference without the mess of the milnerish
constraint system.



From benc at hawaga.org.uk  Sun Jul 24 05:42:02 2011
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 24 Jul 2011 10:42:02 +0000 (GMT)
Subject: [Swift-devel] typed keys
In-Reply-To: <1311502140.10830.20.camel@blabla>
References: <1311362390.23447.11.camel@blabla> 
	<CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com> 
	<1311449444.4930.16.camel@blabla>
	<FC274AF4-A826-4E04-A880-55D982CCF6DF@hawaga.org.uk>
	<1311502140.10830.20.camel@blabla>
Message-ID: <Pine.LNX.4.64.1107241036580.17501@dildano.hawaga.org.uk>


> > introduce runtime type errors: 5+"monkey" (for example) or inability
> > to add two numbers (5+5 = TYPE ERROR! because those 5s happen to be
> > Objects not numbers)
> 
> You would not be able to use these things in any other context than the
> places that accept the any type. Such as trace() and whatever else is
> there that fits, but not "+".

So should this work? O rather, how liberal does compile time type checking 
have to be to allow it to work at runtime?

int a[]; int b[]
populate(a); populate(b); // with number indices
for v,k in a {
 for u,l in b {
  o[k*1000 + l] = f(u,v)
 }
}

Its something that works at the moment. I don't think this style is used a 
huge amount but I think it has been done in the past. Constructing string 
indices by concatenating things is another example of the same, I think.

With a runtime top-type, the compile time checking needs to be quite 
loose, I think - here k and l are used numerically, but if you're keeping 
the discovery of the numericness 'till runtime, then you can't know at 
compile time.

Maybe that is an argument in favour of some compile time inference.

-- 


From iraicu at cs.iit.edu  Sun Jul 24 09:20:47 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sun, 24 Jul 2011 09:20:47 -0500
Subject: [Swift-devel] CFP: 4th ACM Workshop on Many-Task Computing on Grids
 and Supercomputers (MTAGS) 2011 -- co-located with SC11
Message-ID: <4E2C2A3F.50306@cs.iit.edu>


    4th Workshop on Many-Task Computing on Grids and Supercomputers
    (MTAGS) 2011

*Co-located with * *Supercomputing/SC 2011* 
<http://sc11.supercomputing.org/> *
Seattle Washington -- November 14th, 2011*


        Overview

The 4th workshop on Many-Task Computing on Grids and Supercomputers 
(MTAGS) will provide the scientific community a dedicated forum for 
presenting new research, development, and deployment efforts of 
large-scale many-task computing (MTC) applications on large scale 
clusters, Grids, Supercomputers, and Cloud Computing infrastructure. 
MTC, the theme of the workshop encompasses loosely coupled applications, 
which are generally composed of many tasks (both independent and 
dependent tasks) to achieve some larger application goal.  This workshop 
will cover challenges that can hamper efficiency and utilization in 
running applications on large-scale systems, such as local resource 
manager scalability and granularity, efficient utilization of raw 
hardware, parallel file system contention and scalability, data 
management, I/O management, reliability at scale, and application 
scalability. We welcome paper submissions on all topics related to MTC 
on large scale systems.  Papers will be peer-reviewed, and accepted 
papers will be published in the workshop proceedings as part of the ACM 
digital library (pending approval). The workshop will be co-located with 
the IEEE/ACM Supercomputing 2011 Conference in Seattle Washington on 
November 14th, 2011.

For more information, please see 
http://datasys.cs.iit.edu/events/MTAGS11/ 
<http://datasys.cs.iit.edu/events/MTAGS11/index.html>. For more 
information on past workshops, please see MTAGS10 
<http://datasys.cs.iit.edu/events/MTAGS10/index.html>, MTAGS09 
<http://datasys.cs.iit.edu/events/MTAGS09/index.html>, and MTAGS08 
<http://datasys.cs.iit.edu/events/MTAGS08/index.html>. We also ran a 
Special Issue on Many-Task Computing in the IEEE Transactions on 
Parallel and Distributed Systems (TPDS) 
<http://datasys.cs.iit.edu/events/TPDS_MTC/index.html> which has 
appeared in June 2011; the proceedings can be found online at 
http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. 
We, the workshop organizers, also published two papers that are highly 
relevant to this workshop. One paper is titled "Toward Loosely Coupled 
Programming on Petascale Systems 
<http://www.cs.iit.edu/%7Eiraicu/research/publications/2008_SC08_Falkon_BGP.pdf>", 
and was published in SC08 <http://sc08.supercomp.org/>; the second paper 
is titled "Many-Task Computing for Grids and Supercomputers 
<http://www.cs.iit.edu/%7Eiraicu/research/publications/2008_MTAGS08_MTC.pdf>", 
which was published in MTAGS08 
<http://datasys.cs.iit.edu/events/MTAGS08/index.html>.


        Topics

We invite the submission of original work that is related to the topics 
below. The papers can be either short (5 pages) position papers, or long 
(10 pages) research papers. Topics of interest include (in the context 
of Many-Task Computing):

  *   Compute Resource Management
      o Scheduling
      o Job execution frameworks
      o Local resource manager extensions
      o Performance evaluation of resource managers in use on large
        scale systems
      o Dynamic resource provisioning
      o Techniques to manage many-core resources and/or GPUs
      o Challenges and opportunities in running many-task workloads on
        HPC systems
      o Challenges and opportunities in running many-task workloads on
        Cloud Computing infrastructure
  *   Storage architectures and implementations
      o Distributed file systems
      o Parallel file systems
      o Distributed meta-data management
      o Content distribution systems for large data
      o Data caching frameworks and techniques
      o Data management within and across data centers
      o Data-aware scheduling
      o Data-intensive computing applications
      o Eventual-consistency storage usage and management
  *   Programming models and tools
      o Map-reduce and its generalizations
      o Many-task computing middleware and applications
      o Parallel programming frameworks
      o Ensemble MPI techniques and frameworks
      o Service-oriented science applications
  *   Large-Scale Workflow Systems
      o Workflow system performance and scalability analysis
      o Scalability of workflow systems
      o Workflow infrastructure and e-Science middleware
      o Programming Paradigms and Models
  *   Large-Scale Many-Task Applications
      o High-throughput computing (HTC) applications
      o Data-intensive applications
      o Quasi-supercomputing applications, deployments, and experiences
      o Performance Evaluation
  *   Performance evaluation
      o Real systems
      o Simulations
      o Reliability of large systems


        Important Dates

  * Abstract submission: September 2, 2011
  * Paper submission: September 9, 2011
  * Acceptance notification: October 7, 2011
  * Final papers due: October 28, 2011


        Paper Submission

Authors are invited to submit papers with unpublished, original work of 
not more than 10 pages of double column text using single spaced 10 
point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript 
guidelines 
(http://www.acm.org/publications/instructions_for_proceedings_volumes); 
document templates can be found at 
http://www.acm.org/sigs/publications/proceedings-templates. We are also 
seeking position papers of no more than 5 pages in length. A 250 word 
abstract (PDF format) must be submitted online at 
https://cmt.research.microsoft.com/MTAGS2011/ before the deadline of 
September 2nd, 2011 at 11:59PM PST; the final 5/10 page papers in PDF 
format will be due on September 9th, 2011 at 11:59PM PST. Papers will be 
peer-reviewed, and accepted papers will be published in the workshop 
proceedings as part of the ACM digital library (pending approval). 
Notifications of the paper decisions will be sent out by October 7th, 
2011. Selected excellent work may be eligible for additional 
post-conference publication as journal articles or book chapters, such 
as the previous Special Issue on Many-Task Computing in the IEEE 
Transactions on Parallel and Distributed Systems (TPDS) 
<http://datasys.cs.iit.edu/events/TPDS_MTC/index.html> which has 
appeared in June 2011. Submission implies the willingness of at least 
one of the authors to register and present the paper. For more 
information, please http://datasys.cs.iit.edu/events/MTAGS11/ 
<http://datasys.cs.iit.edu/events/MTAGS11/index.html>, or send email to 
mtags11-chairs at datasys.cs.iit.edu 
<mailto:mtags11-chairs at datasys.cs.iit.edu>.


        Organization
        <http://datasys.cs.iit.edu/events/MTAGS11/organization.html>

*General Chairs (mtags11-chairs at datasys.cs.iit.edu 
<mailto:mtags11-chairs at datasys.cs.iit.edu>)*

  * Ioan Raicu, Illinois Institute of Technology & Argonne National
    Laboratory, USA
  * Ian Foster, University of Chicago & Argonne National Laboratory, USA
  * Yong Zhao, University of Electronic Science and Technology of China,
    China

*Steering Committee*

  * David Abramson, Monash University, Australia
  * Jack Dongara, University of Tennessee, USA
  * Geoffrey Fox, Indiana University, USA
  * Xian-He Sun, Illinois Institute of Technology, USA

*Program Committee*

  * Roger Barga, Microsoft Research, USA
  * Mihai Budiu, Microsoft Research, USA
  * Rajkumar Buyya, University of Melbourne, Australia
  * Catalin Dumitrescu, Fermi National Labs, USA
  * Alexandru Iosup, Delft University of Technology, Netherlands
  * Florin Isaila, Universidad Carlos III de Madrid, Spain
  * Kamil Iskra, Argonne National Laboratory, USA
  * Hui Jin, Illinois Institute of Technology, USA
  * Daniel S. Katz, University of Chicago, USA
  * Tevfik Kosar, Louisiana State University, USA
  * Zhiling Lan, Illinois Institute of Technology, USA
  * Reagan Moore, University of North Carolina, Chappel Hill, USA
  * Jose Moreira, IBM Research, USA
  * Judy Qui, Indiana University, USA
  * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
  * Matei Ripeanu, University of British Columbia, Canada
  * Alain Roy, University of Wisconsin, Madison, USA
  * Edward Walker, Texas Advanced Computing Center, USA
  * Matthew Woitaszek, The University Corporation for Atmospheric
    Research, USA

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110724/2642e93f/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 24 14:34:20 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 24 Jul 2011 12:34:20 -0700
Subject: [Swift-devel] typed keys
In-Reply-To: <Pine.LNX.4.64.1107241036580.17501@dildano.hawaga.org.uk>
References: <1311362390.23447.11.camel@blabla>
	<CANa904mbfKPiZ-sEKeLZtaid0Nd6gC6pmFHLXLMWYt25rqLotA@mail.gmail.com>
	<1311449444.4930.16.camel@blabla>
	<FC274AF4-A826-4E04-A880-55D982CCF6DF@hawaga.org.uk>
	<1311502140.10830.20.camel@blabla>
	<Pine.LNX.4.64.1107241036580.17501@dildano.hawaga.org.uk>
Message-ID: <1311536060.11978.8.camel@blabla>

On Sun, 2011-07-24 at 10:42 +0000, Ben Clifford wrote:
> > > introduce runtime type errors: 5+"monkey" (for example) or inability
> > > to add two numbers (5+5 = TYPE ERROR! because those 5s happen to be
> > > Objects not numbers)
> > 
> > You would not be able to use these things in any other context than the
> > places that accept the any type. Such as trace() and whatever else is
> > there that fits, but not "+".
> 
> So should this work? O rather, how liberal does compile time type checking 
> have to be to allow it to work at runtime?
> 
> int a[]; int b[]
> populate(a); populate(b); // with number indices
> for v,k in a {
>  for u,l in b {
>   o[k*1000 + l] = f(u,v)
>  }
> }

Yeah, this should work. It would also allow mixed types:
a[1] = 1; a["b"] = 2;

You won't be able to say:
f(int) {}
foreach v, k in a {
  f(k);
}

But that is fine. If you want that, you can declare a as int[int], and
then you can't have a["b"].

Though this issue is purely hypothetical at this point. It's not the
solution I would favor.

What I favor is auto. We're basically adding the append, so to keep it
separate from everything else we have so far, we would associate it with
the auto type.

> 
> Its something that works at the moment. I don't think this style is used a 
> huge amount but I think it has been done in the past. Constructing string 
> indices by concatenating things is another example of the same, I think.
> 
> With a runtime top-type, the compile time checking needs to be quite 
> loose, I think - here k and l are used numerically, but if you're keeping 
> the discovery of the numericness 'till runtime, then you can't know at 
> compile time.

Again, you wouldn't be able to use an "any" as an int.

> 
> Maybe that is an argument in favour of some compile time inference.
> 

Loosely.



From hategan at mcs.anl.gov  Sun Jul 24 18:18:56 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 24 Jul 2011 16:18:56 -0700
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <CANa904nBj1w6DZo-0x+E8a=AdDNuSuSvru2q8rdtfi=qScLK-g@mail.gmail.com>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
	<CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
	<1311101765.20818.1.camel@blabla>
	<CANa904nGOhCc6y7_V4yTbLJ4u1gSJP8z2NN1XSFM-r0Bsh7Hbg@mail.gmail.com>
	<CANa904nBj1w6DZo-0x+E8a=AdDNuSuSvru2q8rdtfi=qScLK-g@mail.gmail.com>
Message-ID: <1311549536.13484.13.camel@blabla>

This is now in trunk after a few modifications.
What's to be noted about key types (an probably put in some
documentation):
1. Appendable arrays need to be declared like this: a[auto]. "auto" is a
valid type, in that one can do the following:
int[auto] a; a << 1; a << 2;
int[auto] b;
foreach v, k in a {
  auto k2 = k;
  b[k2] = v;
}

2. a[] means a[int].

3. When mapping keys the standard mappers do the following to get the
part of a filename corresponding to a key:
 - for int keys, behave as previously (i.e. use the value)
 - for float keys, use the hex representation of the double value
 - for any other keys, base64(SHA-1(<the string representation of the
object>))

The problem of representing arbitrary-typed keys remains, especially
when stepping out of the primitive area.
------
I've also changed the following:
- the internal representation of an int value now uses java.lang.Integer
as opposed to java.lang.Double.
- arrays can now be declared properly: int[] intArray vs. int
intArray[]. But the old method still works.
- a recursive-descent parser has been added for paths (due to the
complexities of now being able to use floats and strings as keys).

All local tests pass.

So  thanks Yadu!

Mihael

On Wed, 2011-07-20 at 11:39 +0530, Yadu Nand wrote:
> > Great!
> > Please find the patch attached. I see that there are a lot of changes which
> > are actually due to some extra white-spaces (possibly), which will make
> > reading the actual diff a bit difficult.
> My bad, forgot the patch again. :(
> 




From iraicu at cs.iit.edu  Sun Jul 24 23:37:36 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sun, 24 Jul 2011 23:37:36 -0500
Subject: [Swift-devel] CFP: The Second ACM International Workshop on Data
 Intensive Computing in the Clouds (DataCloud-SC11) 2011 -- co-located with
 SC11
Message-ID: <4E2CF310.5050403@cs.iit.edu>


    The Second International Workshop on Data Intensive Computing in the
    Clouds (DataCloud-SC11) 2011

http://datasys.cs.iit.edu/events/DataCloud-SC11/

*Co-located with * *Supercomputing/SC 2011* 
<http://sc11.supercomputing.org/> *
Seattle Washington -- November 14th, 2011*


        Overview

Applications and experiments in all areas of science are becoming 
increasingly complex and more demanding in terms of their computational 
and data requirements. Some applications generate data volumes reaching 
hundreds of terabytes and even petabytes. As scientific applications 
become more data intensive, the management of data resources and 
dataflow between the storage and compute resources is becoming the main 
bottleneck. Analyzing, visualizing, and disseminating these large data 
sets has become a major challenge and data intensive computing is now 
considered as the "fourth paradigm" in scientific discovery after 
theoretical, experimental, and computational science.

The second international workshop on Data-intensive Computing in the 
Clouds (DataCloud-SC11) will provide the scientific community a 
dedicated forum for discussing new research, development, and deployment 
efforts in running data-intensive computing workloads on Cloud Computing 
infrastructures. The DataCloud-SC11 workshop will focus on the use of 
cloud-based technologies to meet the new data intensive scientific 
challenges that are not well served by the current supercomputers, grids 
or compute-intensive clouds. We believe the workshop will be an 
excellent place to help the community define the current state, 
determine future goals, and present architectures and services for 
future clouds supporting data intensive computing.

For more information about the workshop, please see 
http://datasys.cs.iit.edu/events/DataCloud-SC11/. To see the 1st 
workshop's program agenda, and accepted papers and presentations, please 
see http://www.cse.buffalo.edu/faculty/tkosar/datacloud2011/. We are 
also running a Special Issue on Data Intensive Computing in the Clouds 
<http://datasys.cs.iit.edu/events/JGC-DataCloud-2012/index.html> in the 
Springer Journal of Grid Computing 
<http://www.springer.com/computer/communication+networks/journal/10723> 
with a paper submission deadline of August 16th 2011, which will appear 
in print in June 2012.


        Topics

  * Data-intensive cloud computing applications, characteristics,
    challenges
  * Case studies of data intensive computing in the clouds
  * Performance evaluation of data clouds, data grids, and data centers
  * Energy-efficient data cloud design and management
  * Data placement, scheduling, and interoperability in the clouds
  * Accountability, QoS, and SLAs
  * Data privacy and protection in a public cloud environment
  * Distributed file systems for clouds
  * Data streaming and parallelization
  * New programming models for data-intensive cloud computing
  * Scalability issues in clouds
  * Social computing and massively social gaming
  * 3D Internet and implications
  * Future research challenges in data-intensive cloud computing


        Important Dates

  * Abstract submission: September 2, 2011
  * Paper submission: September 9, 2011
  * Acceptance notification: October 7, 2011
  * Final papers due: October 28, 2011


        Paper Submission

Authors are invited to submit papers with unpublished, original work of 
not more than 10 pages of double column text using single spaced 10 
point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript 
guidelines 
(http://www.acm.org/publications/instructions_for_proceedings_volumes); 
document templates can be found at 
http://www.acm.org/sigs/publications/proceedings-templates. We are also 
seeking position papers of no more than 5 pages in length. A 250 word 
abstract (PDF format) must be submitted online at 
https://cmt.research.microsoft.com/DataCloud_SC11/ before the deadline 
of September 2nd, 2011 at 11:59PM PST; the final 5/10 page papers in PDF 
format will be due on September 9th, 2011 at 11:59PM PST. Papers will be 
peer-reviewed, and accepted papers will be published in the workshop 
proceedings as part of the ACM digital library (pending approval). 
Notifications of the paper decisions will be sent out by October 7th, 
2011. Selected excellent work may be eligible for additional 
post-conference publication as journal articles. We are currently 
running a Special Issue on Data Intensive Computing in the Clouds 
<http://datasys.cs.iit.edu/events/JGC-DataCloud-2012/index.html> in the 
Springer Journal of Grid Computing 
<http://www.springer.com/computer/communication+networks/journal/10723>. 
Submission implies the willingness of at least one of the authors to 
register and present the paper. For more information, please see 
http://datasys.cs.iit.edu/events/DataCloud-SC11/ or send email to 
datacloud-sc11-chairs at datasys.cs.iit.edu 
<mailto:datacloud-sc11-chairs at datasys.cs.iit.edu>.


        Organization
        <http://datasys.cs.iit.edu/events/DataCloud-SC11/organization.html>

*General Chairs (datacloud-sc11-chairs at datasys.cs.iit.edu 
<mailto:datacloud-sc11-chairs at datasys.cs.iit.edu>)*

  * Ioan Raicu, Illinois Institute of Technology & Argonne National
    Laboratory, USA
  * Tevfik Kosar,  University at Buffalo, USA
  * Roger Barga, Microsoft Research, USA

*Steering Committee*

  * Ian Foster, University of Chicago & Argonne National Laboratory, USA
  * Geoffrey Fox, Indiana University, USA
  * James Hamilton, Amazon, USA
  * Manish Parashar, Rutgers University, USA
  * Dan Reed, Microsoft Research, USA
  * Rich Wolski, University of California at Santa Barbara, USA
  * Liang-Jie Zhang, IBM T.J. Watson Research Center, USA

*Program Committee*

  * David Abramson, Monash University, Australia
  * Abhishek Chandra, University of Minnesota, USA
  * Rong Chang, IBM, USA
  * Yong Chen, Texas Tech University, USA
  * Terence Critchlow, Pacific Northwest National Laboratory, USA
  * Murat Demirbas, SUNY Buffalo, USA
  * Jaliya Ekanayake, Microsoft Research, USA
  * Rob Gillen, Oak Ridge National Laboratory, USA
  * Maria Indrawan, Monash University, Australia
  * Alexandru Iosup, Delft University of Technology, Netherlands
  * Hui Jin, Illinois Institute of Technology, USA
  * Dan S. Katz, University of Chicago, USA
  * Gregor von Laszewski, Indiana University, USA
  * Erwin Laure, CERN, Switzerland
  * Reagan Moore, University of North Carolina at Chapel Hill, USA
  * Judy Qiu, Indiana University, USA
  * Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
  * Florian Schintke, Zuse Institute Berlin, Germany
  * Borja Sotomayor, University of Chicago, USA
  * Ian Taylor, Cardiff University, UK
  * Bernard Traversat, Oracle Corporation, USA

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:  iraicu at cs.iit.edu
Web:    http://www.cs.iit.edu/~iraicu/
Web:    http://datasys.cs.iit.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110724/e9a93661/attachment.html>

From yadudoc1729 at gmail.com  Mon Jul 25 00:46:18 2011
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Mon, 25 Jul 2011 11:16:18 +0530
Subject: [Swift-devel] Updates - Tests
In-Reply-To: <1311549536.13484.13.camel@blabla>
References: <CANa904k_TPNxrTXj6EjeQw7JQQhcWUQ7prDOXdPSXm1KS+KkGg@mail.gmail.com>
	<1310671184.10910.4.camel@blabla>
	<CANa904n7RyyY=+KymDK4xvh32mdkvCtEVRjPOTK66rabtHet2Q@mail.gmail.com>
	<1310674982.11160.0.camel@blabla>
	<CANa904=e3yOLdSU=SeOVWWtntPOLLxDyYC-UMcg13cJnuVEVBA@mail.gmail.com>
	<1310682629.12137.1.camel@blabla>
	<CANa904=gSbf8ZFFdsU+Onwdv40itVS=pM_20E29_uYXULw6EZQ@mail.gmail.com>
	<1311101765.20818.1.camel@blabla>
	<CANa904nGOhCc6y7_V4yTbLJ4u1gSJP8z2NN1XSFM-r0Bsh7Hbg@mail.gmail.com>
	<CANa904nBj1w6DZo-0x+E8a=AdDNuSuSvru2q8rdtfi=qScLK-g@mail.gmail.com>
	<1311549536.13484.13.camel@blabla>
Message-ID: <CANa904=iiKWp5G7wgZCTji4FO1ux+_dFA51fXMCxernyEFaPHQ@mail.gmail.com>

> I've also changed the following:
> - the internal representation of an int value now uses java.lang.Integer
> as opposed to java.lang.Double.
> - arrays can now be declared properly: int[] intArray vs. int
> intArray[]. But the old method still works.
> - a recursive-descent parser has been added for paths (due to the
> complexities of now being able to use floats and strings as keys).
> All local tests pass.
Yay!

> So ?thanks Yadu!
Always welcome :)

-- 
Thanks and Regards,
Yadu Nand B


From wozniak at mcs.anl.gov  Tue Jul 26 10:34:45 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Tue, 26 Jul 2011 10:34:45 -0500 (CDT)
Subject: [Swift-devel] Fwd: 2011 summer intern tour - Thur., 7/28 (fwd)
Message-ID: <alpine.DEB.2.00.1107261033320.12716@wozniak-desktop.mcs.anl.gov>


ALCF tours for summer students.  If interested, you have to sign up ASAP.

-- 
Justin M Wozniak

---------- Forwarded message ----------
Date: Tue, 26 Jul 2011 10:26:26
From: Jason Cope <copej at mcs.anl.gov>
To: Justin Wozniak <wozniak at mcs.anl.gov>
Subject: Fwd: 2011 summer intern tour - Thur., 7/28

Begin forwarded message:

> From: Susan Gregurich <gregurich at anl.gov>
> Date: July 25, 2011 5:30:42 PM CDT
> To:

> Did you know the Argonne Leadership Computing Facility houses one of the 
> fastest supercomputers on the planet--the IBM Blue Gene/P, called Intrepid? 
> Join us for a tour of our Interim Supercomputing Support Facility (ISSF) on 
> Thursday, July 28th.
> 
> We are offering tours at 1-1:45 p.m. for Group 1 and 2-2:45 p.m. for Group 2. 
> There is a 20-person max. limit per tour group. Spaces are limited so please 
> RSVP by replying to this email as soon as possible with the time slot you 
> would prefer. We will make every effort to accommodate your first selection. 
> Registration cut-off is Wednesday, 7/27/11.
> 
> Catch a shuttle bus from the main entrance of building 240 (near Fed Ex box): 
> departure times of 12:50 p.m. for Group 1 and for Group 2 at 1:50 p.m. (or as 
> soon as the bus returns with Group 1 from building 369).
> 
> Note:
> - Heels are not permitted in the ISSF so please do not wear them to the tour
> - Still photos are permitted



From ketancmaheshwari at gmail.com  Tue Jul 26 11:03:39 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 26 Jul 2011 11:03:39 -0500
Subject: [Swift-devel] How Swift treats gridftp'd data on local
Message-ID: <CAMUuvir1c_kk7PzQ161Coe6L5oDs_rxA-vjbDc-rJ+F6mTVkqg@mail.gmail.com>

Hi All,

I was wondering if Swift treats a gridtftp'd piece of data as *local* that
is destined to be consumed on the same host as the running gridftp server?

I am trying to run some of the SCEC workflow on ranger resources where the
data is also hosted on its gridftp server. The data staging is apparantly
slow and log does not seem to be showing much info on this. I do see the
gridftp URLs on the staging lines though.

The log is :
http://www.mcs.anl.gov/~ketan/files/postproc-20110725-2345-k1mi19f7.log(1035 MB)

Regards,
-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110726/338435c1/attachment.html>

From ketancmaheshwari at gmail.com  Tue Jul 26 11:58:28 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Tue, 26 Jul 2011 11:58:28 -0500
Subject: [Swift-devel] How Swift treats gridftp'd data on local
In-Reply-To: <CAMUuvir1c_kk7PzQ161Coe6L5oDs_rxA-vjbDc-rJ+F6mTVkqg@mail.gmail.com>
References: <CAMUuvir1c_kk7PzQ161Coe6L5oDs_rxA-vjbDc-rJ+F6mTVkqg@mail.gmail.com>
Message-ID: <CAMUuvio7-qeqsnceMwoDZPyo2-z+FM08tPBrOLAaJtxLFb00Dg@mail.gmail.com>

In addition, I see in the log lines corresponding to staging:
policy=DEFAULT, wondering if it has to do with the decision on how the
staging takes place.


On Tue, Jul 26, 2011 at 11:03 AM, Ketan Maheshwari <
ketancmaheshwari at gmail.com> wrote:

> Hi All,
>
> I was wondering if Swift treats a gridtftp'd piece of data as *local* that
> is destined to be consumed on the same host as the running gridftp server?
>
> I am trying to run some of the SCEC workflow on ranger resources where the
> data is also hosted on its gridftp server. The data staging is apparantly
> slow and log does not seem to be showing much info on this. I do see the
> gridftp URLs on the staging lines though.
>
> The log is :
> http://www.mcs.anl.gov/~ketan/files/postproc-20110725-2345-k1mi19f7.log(1035 MB)
>
> Regards,
> --
> Ketan
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110726/180957d3/attachment.html>

From aespinosa at cs.uchicago.edu  Tue Jul 26 12:11:21 2011
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Tue, 26 Jul 2011 11:11:21 -0600
Subject: [Swift-devel] How Swift treats gridftp'd data on local
In-Reply-To: <CAMUuvir1c_kk7PzQ161Coe6L5oDs_rxA-vjbDc-rJ+F6mTVkqg@mail.gmail.com>
References: <CAMUuvir1c_kk7PzQ161Coe6L5oDs_rxA-vjbDc-rJ+F6mTVkqg@mail.gmail.com>
Message-ID: <CAAHfuQ9Dm5wFpsK_dOcjH6WpRPECx3m9xi77x07=-WJs=BS16g@mail.gmail.com>

Unless the site-specific CDM directives are made, I don't think you
can do that (yet).

2011/7/26 Ketan Maheshwari <ketancmaheshwari at gmail.com>:
> Hi All,
>
> I was wondering if Swift treats a gridtftp'd piece of data as *local* that
> is destined to be consumed on the same host as the running gridftp server?
>
> I am trying to run some of the SCEC workflow on ranger resources where the
> data is also hosted on its gridftp server. The data staging is apparantly
> slow and log does not seem to be showing much info on this. I do see the
> gridftp URLs on the staging lines though.
>
> The log is :
> http://www.mcs.anl.gov/~ketan/files/postproc-20110725-2345-k1mi19f7.log
> (1035 MB)
>
> Regards,
> --
> Ketan
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From hategan at mcs.anl.gov  Wed Jul 27 00:45:56 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 26 Jul 2011 22:45:56 -0700
Subject: [Swift-devel] measuring queue times
Message-ID: <1311745556.13580.6.camel@blabla>

So I've got a small tool to do it: www.mcs.anl.gov/~hategan/qt.tar.gz

Run like this:
./qtest count=128 range=240 parallelism=1 provider=gt2 jobmanager=sge
host=gatekeeper.ranger.tacc.teragrid.org

You can also pass various job attributes in the same way (e.g.
"queue=long")

Parallelism should be kept at 1 (one job at a time). The range is the
walltime range in minutes. The actual walltimes are randomly generated
within that range. Count is the total count (there is no penalty in
having a large number there).

I'm running this on ranger, and pads seems to have a bug where jobs are
reported running by pbs and then they go back to being queued. But
please run this on other systems if you can and send me back the
resulting queuetest.out. You can just check to see if it works with one
job and then let it run unsupervised in a screen session or so.



From ketancmaheshwari at gmail.com  Wed Jul 27 15:00:36 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Wed, 27 Jul 2011 15:00:36 -0500
Subject: [Swift-devel] measuring queue times
In-Reply-To: <1311745556.13580.6.camel@blabla>
References: <1311745556.13580.6.camel@blabla>
Message-ID: <CAMUuvipUj=Cp52tYmRCj085oauBsm+kGiX74dYsxESkPNc_6OA@mail.gmail.com>

Mihael,

Attached is the output and log for beagle. Seems, the first few lines in the
out file seems to be from my trial commandline and could be stripped off.

Following is a snapshot of the run:

./qtest count=128 range=240 parallelism=1 provider=pbs jobmanager=pbs host=
login1.beagle.ci.uchicago.edu
.:cog-abstraction-common-2.4.jar:cog-jglobus-1.8.0.jar:cog-provider-gt2-2.4.jar:cog-provider-local-2.2.jar:cog-provider-localscheduler-0.4.jar:cog-util-0.92.jar:commons-logging-1.1.jar:cryptix32.jar:cryptix-asn1.jar:cryptix.jar:jce-jdk13-131.jar:jgss.jar:log4j-1.2.16.jar:puretls.jar
2015328287 submitting (1)
2015328287 running
2015328287 195 10
2015328287 done
1182680702 submitting (2)
1182680702 running
1182680702 48 18
1182680702 done
2057140893 submitting (3)
2057140893 running
2057140893 2 9
...
...


How many runs per cluster you are thinking of?


Regards,
Ketan


On Wed, Jul 27, 2011 at 12:45 AM, Mihael Hategan <hategan at mcs.anl.gov>wrote:

> So I've got a small tool to do it: www.mcs.anl.gov/~hategan/qt.tar.gz<http://www.mcs.anl.gov/%7Ehategan/qt.tar.gz>
>
> Run like this:
> ./qtest count=128 range=240 parallelism=1 provider=gt2 jobmanager=sge
> host=gatekeeper.ranger.tacc.teragrid.org
>
> You can also pass various job attributes in the same way (e.g.
> "queue=long")
>
> Parallelism should be kept at 1 (one job at a time). The range is the
> walltime range in minutes. The actual walltimes are randomly generated
> within that range. Count is the total count (there is no penalty in
> having a large number there).
>
> I'm running this on ranger, and pads seems to have a bug where jobs are
> reported running by pbs and then they go back to being queued. But
> please run this on other systems if you can and send me back the
> resulting queuetest.out. You can just check to see if it works with one
> job and then let it run unsupervised in a screen session or so.
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110727/62caa5a9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: queuetest.out
Type: application/octet-stream
Size: 6629 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110727/62caa5a9/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.log
Type: application/octet-stream
Size: 399854 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110727/62caa5a9/attachment-0001.obj>

From hategan at mcs.anl.gov  Wed Jul 27 15:52:19 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 27 Jul 2011 13:52:19 -0700
Subject: [Swift-devel] measuring queue times
In-Reply-To: <CAMUuvipUj=Cp52tYmRCj085oauBsm+kGiX74dYsxESkPNc_6OA@mail.gmail.com>
References: <1311745556.13580.6.camel@blabla>
	<CAMUuvipUj=Cp52tYmRCj085oauBsm+kGiX74dYsxESkPNc_6OA@mail.gmail.com>
Message-ID: <1311799939.19191.2.camel@blabla>

On Wed, 2011-07-27 at 15:00 -0500, Ketan Maheshwari wrote:
> Mihael,
> 
> Attached is the output and log for beagle. Seems, the first few lines
> in the out file seems to be from my trial commandline and could be
> stripped off.

Thanks.

[...]
> 
> How many runs per cluster you are thinking of?

This will probably do. The only think might be to increase the range as
far as the queue supports.

Mihael



From wozniak at mcs.anl.gov  Wed Jul 27 15:53:51 2011
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Wed, 27 Jul 2011 15:53:51 -0500 (Central Daylight Time)
Subject: [Swift-devel] Call today
Message-ID: <alpine.WNT.2.00.1107271553110.3040@justinwozniak>


I'll be on for our regular call today.

-- 
Justin M Wozniak


From iraicu at cs.iit.edu  Wed Jul 27 16:55:33 2011
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Wed, 27 Jul 2011 16:55:33 -0500
Subject: [Swift-devel] Fwd: Journal of Grid Computing has got impact factor
	1.556
In-Reply-To: <Pine.LNX.4.64.1107182258150.6405@vidra.sztaki.hu>
References: <Pine.LNX.4.64.1107182258150.6405@vidra.sztaki.hu>
Message-ID: <4E308955.9000302@cs.iit.edu>

FYI

-------- Original Message --------
Subject: 	Journal of Grid Computing has got impact factor 1.556
Date: 	Mon, 18 Jul 2011 23:11:57 +0200 (CEST)
From: 	Kacsuk Peter <kacsuk at sztaki.hu>
To:
	iraicu at cs.iit.edu



Dear All,

As the previous editors of JOGC there is a good news for you: The 2010
Impact Factors have just been released and JOGC is included first time. It
has an impact factor of 1.556 and is ranked 43/126 journals in Computer
Science/Information Systems and 29/97 in Computer Science/Theory and
Methods.

So not bad at all for a first ranking.

It is worth to have a look at the following web page where Springer shows
the journals of High Impact Factors - Increased Impact Factors - First
Impact Factors

http://www.springer.com/computer/computer+journals?SGWID=0-40100-12-836504-0

Here you can see that JOGC has got the highest IF among the journals of
First Impact Factors and ranked higher than many Springer journals that
had got IF earlier.

Many thanks for your contribution to achieve this outstanding performance.

May I ask you to distribute this information to grid and cloud
communities/projects that you have got contact with and invite them to
submit more articles in the future? I think that this impact factor makes
JOGC attractive enough for many researchers to publish papers in it.

Regards,
Peter


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110727/45c2611c/attachment.html>

From ketancmaheshwari at gmail.com  Wed Jul 27 17:37:47 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Wed, 27 Jul 2011 17:37:47 -0500
Subject: [Swift-devel] measuring queue times
In-Reply-To: <1311799939.19191.2.camel@blabla>
References: <1311745556.13580.6.camel@blabla>
	<CAMUuvipUj=Cp52tYmRCj085oauBsm+kGiX74dYsxESkPNc_6OA@mail.gmail.com>
	<1311799939.19191.2.camel@blabla>
Message-ID: <CAMUuvir771RrDm45-dCJeE4V+ihy9Jo+xy8V+rxuCU81N4RA_A@mail.gmail.com>

I think the maximum walltime that the queue supports on Beagle is 48 hours.
I will put a run with this time in a screen session.


On Wed, Jul 27, 2011 at 3:52 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Wed, 2011-07-27 at 15:00 -0500, Ketan Maheshwari wrote:
> > Mihael,
> >
> > Attached is the output and log for beagle. Seems, the first few lines
> > in the out file seems to be from my trial commandline and could be
> > stripped off.
>
> Thanks.
>
> [...]
> >
> > How many runs per cluster you are thinking of?
>
> This will probably do. The only think might be to increase the range as
> far as the queue supports.
>
> Mihael
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110727/002a3c5d/attachment.html>

From ketancmaheshwari at gmail.com  Thu Jul 28 09:50:34 2011
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 28 Jul 2011 09:50:34 -0500
Subject: [Swift-devel] measuring queue times
In-Reply-To: <CAMUuvir771RrDm45-dCJeE4V+ihy9Jo+xy8V+rxuCU81N4RA_A@mail.gmail.com>
References: <1311745556.13580.6.camel@blabla>
	<CAMUuvipUj=Cp52tYmRCj085oauBsm+kGiX74dYsxESkPNc_6OA@mail.gmail.com>
	<1311799939.19191.2.camel@blabla>
	<CAMUuvir771RrDm45-dCJeE4V+ihy9Jo+xy8V+rxuCU81N4RA_A@mail.gmail.com>
Message-ID: <CAMUuvioPxSXm1ucehhgKaxcmLjjnLkfJv-kXxeRt6T6d+QrA4Q@mail.gmail.com>

Attached is the queuetest.out for 48 hours walltime range.

On Wed, Jul 27, 2011 at 5:37 PM, Ketan Maheshwari <
ketancmaheshwari at gmail.com> wrote:

>
> I think the maximum walltime that the queue supports on Beagle is 48 hours.
> I will put a run with this time in a screen session.
>
>
>
> On Wed, Jul 27, 2011 at 3:52 PM, Mihael Hategan <hategan at mcs.anl.gov>wrote:
>
>> On Wed, 2011-07-27 at 15:00 -0500, Ketan Maheshwari wrote:
>> > Mihael,
>> >
>> > Attached is the output and log for beagle. Seems, the first few lines
>> > in the out file seems to be from my trial commandline and could be
>> > stripped off.
>>
>> Thanks.
>>
>> [...]
>> >
>> > How many runs per cluster you are thinking of?
>>
>> This will probably do. The only think might be to increase the range as
>> far as the queue supports.
>>
>> Mihael
>>
>>
>
>
> --
> Ketan
>
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110728/5616c800/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: queuetest.out
Type: application/octet-stream
Size: 12419 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110728/5616c800/attachment.obj>

From hategan at mcs.anl.gov  Sat Jul 30 13:36:34 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 30 Jul 2011 11:36:34 -0700
Subject: [Swift-devel] gridftp UI
Message-ID: <1312050994.11641.9.camel@blabla>

I needed something to quickly play with files on ranger, and since I
lost my password (again) and I can't seem to get gsissh to work, I
needed some form of gridftp UI.

Cog has (had) a crappy one which still breaks in two many ways. So I
wrote a quick plugin for a tool named muCommander.

So here's the binary (run with java -jar muc.jar):
www.mcs.anl.gov/~hategan/muc/muc.jar

And since it's GPL, here are the sources:
www.mcs.anl.gov/~hategan/muc/mucommander.tar.gz



From foster at anl.gov  Sat Jul 30 13:49:47 2011
From: foster at anl.gov (Ian Foster)
Date: Sat, 30 Jul 2011 13:49:47 -0500
Subject: [Swift-devel] gridftp UI
In-Reply-To: <1312050994.11641.9.camel@blabla>
References: <1312050994.11641.9.camel@blabla>
Message-ID: <242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>

Why not use Globus Online?

On Jul 30, 2011, at 1:36 PM, Mihael Hategan wrote:

> I needed something to quickly play with files on ranger, and since I
> lost my password (again) and I can't seem to get gsissh to work, I
> needed some form of gridftp UI.
> 
> Cog has (had) a crappy one which still breaks in two many ways. So I
> wrote a quick plugin for a tool named muCommander.
> 
> So here's the binary (run with java -jar muc.jar):
> www.mcs.anl.gov/~hategan/muc/muc.jar
> 
> And since it's GPL, here are the sources:
> www.mcs.anl.gov/~hategan/muc/mucommander.tar.gz
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From hategan at mcs.anl.gov  Sat Jul 30 14:54:19 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 30 Jul 2011 12:54:19 -0700
Subject: [Swift-devel] gridftp UI
In-Reply-To: <242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>
References: <1312050994.11641.9.camel@blabla>
	<242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>
Message-ID: <1312055659.11641.33.camel@blabla>

On Sat, 2011-07-30 at 13:49 -0500, Ian Foster wrote:
> Why not use Globus Online?

Convenience/usability. I hit F3 or enter on coasters.log, and it quickly
pops up in a viewer/editor. I hit F8 and the file gets deleted. When I
copy files, I get immediate feedback (in particular progress with
estimated times and speed and everything).

Otherwise GO is pretty cute.




From hategan at mcs.anl.gov  Sat Jul 30 20:35:38 2011
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 30 Jul 2011 18:35:38 -0700
Subject: [Swift-devel] gridftp UI
In-Reply-To: <1312055659.11641.33.camel@blabla>
References: <1312050994.11641.9.camel@blabla>
	<242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>
	<1312055659.11641.33.camel@blabla>
Message-ID: <1312076138.15148.11.camel@blabla>

On Sat, 2011-07-30 at 12:54 -0700, Mihael Hategan wrote:
> On Sat, 2011-07-30 at 13:49 -0500, Ian Foster wrote:
> > Why not use Globus Online?
> 
> Convenience/usability. I hit F3 or enter on coasters.log, and it quickly
> pops up in a viewer/editor. I hit F8 and the file gets deleted. When I
> copy files, I get immediate feedback (in particular progress with
> estimated times and speed and everything).
> 
> Otherwise GO is pretty cute.

While we're here, I don't see any good reason why Globus Online couldn't
support similar features (e.g., delete files, create directories,
view/edit files, better feedback on transfers, etc.). It might even go
as far as starting jobs on the endpoints.

And the agent could support uploading a proxy directly so that getting a
credential would take one step less. It would even be more secure to not
have to go through a third-party (one less thing to trust).




From jonmon at mcs.anl.gov  Sat Jul 30 20:37:52 2011
From: jonmon at mcs.anl.gov (Jonathan Monette)
Date: Sat, 30 Jul 2011 20:37:52 -0500
Subject: [Swift-devel] gridftp UI
In-Reply-To: <1312076138.15148.11.camel@blabla>
References: <1312050994.11641.9.camel@blabla>
	<242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>
	<1312055659.11641.33.camel@blabla>
	<1312076138.15148.11.camel@blabla>
Message-ID: <8A181494-F88C-4372-A99B-64F7D940D26F@mcs.anl.gov>


On Jul 30, 2011, at 8:35 PM, Mihael Hategan wrote:

> On Sat, 2011-07-30 at 12:54 -0700, Mihael Hategan wrote:
>> On Sat, 2011-07-30 at 13:49 -0500, Ian Foster wrote:
>>> Why not use Globus Online?
>> 
>> Convenience/usability. I hit F3 or enter on coasters.log, and it quickly
>> pops up in a viewer/editor. I hit F8 and the file gets deleted. When I
>> copy files, I get immediate feedback (in particular progress with
>> estimated times and speed and everything).
>> 
>> Otherwise GO is pretty cute.
> 
> While we're here, I don't see any good reason why Globus Online couldn't
> support similar features (e.g., delete files, create directories,
> view/edit files, better feedback on transfers, etc.). It might even go
> as far as starting jobs on the endpoints.
> 
> And the agent could support uploading a proxy directly so that getting a
> credential would take one step less. It would even be more secure to not
> have to go through a third-party (one less thing to trust).
You can already upload a proxy that you have created.  Most of the GO commands accept a -g option which gets the credential that was used when you use gsissh.
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From foster at anl.gov  Sun Jul 31 14:52:32 2011
From: foster at anl.gov (Ian Foster)
Date: Sun, 31 Jul 2011 14:52:32 -0500
Subject: [Swift-devel] gridftp UI
In-Reply-To: <1312055659.11641.33.camel@blabla>
References: <1312050994.11641.9.camel@blabla>
	<242183F3-6AFB-4223-8E4A-50D7665C253B@anl.gov>
	<1312055659.11641.33.camel@blabla>
Message-ID: <A63BBFE2-FA24-40E8-8EB3-68D785642EC9@anl.gov>

Understood. We want to add those more interactive capabilities, but it will be a while.

Thanks for the feedback.

On Jul 30, 2011, at 2:54 PM, Mihael Hategan wrote:

> On Sat, 2011-07-30 at 13:49 -0500, Ian Foster wrote:
>> Why not use Globus Online?
> 
> Convenience/usability. I hit F3 or enter on coasters.log, and it quickly
> pops up in a viewer/editor. I hit F8 and the file gets deleted. When I
> copy files, I get immediate feedback (in particular progress with
> estimated times and speed and everything).
> 
> Otherwise GO is pretty cute.
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From etsamour at csd.auth.gr  Fri Jul  8 07:26:33 2011
From: etsamour at csd.auth.gr (Efthymia Tsamoura)
Date: Fri, 08 Jul 2011 12:26:33 -0000
Subject: [Swift-devel] extending swift with workflow optimization algorithms
Message-ID: <20110708152527.12700w6kenz5kbav@webmail.auth.gr>

Hello

I am a phd student and during this period i am dealing with workflow  
optimization problems in distributed environments.  I would like to  
ask, if there are exist any cases where if the order of task  
invocation in a scientific workflow changes its performance changes  
too without, however, affecting the produced results. In the  
following, a present a small use case of the problem i am interested in:

Suppose that a company wants to obtain a list of email addresses of  
potential customers selecting only those who have a good payment  
history for at least one card and a credit rating above some  
threshold. The company has the right to use the following web services

WS1 : SSN id (ssn, threshold) -> credit rating (cr)
WS2 : SSN id (ssn) -> credit card numbers (ccn)
WS3 : card number (ccn, good) -> good history (gph)
WS4 : SSN id (ssn) -> email addresses (ea)

The input data containing customer identifiers (ssn) and other  
relevant information is stored in a local data resource. Two possible  
web service linear workflows that can be formed to process the input  
data using the above services are C1 = WS2,WS3,WS1,WS4 and C2 =  
WS1,WS2,WS3,WS4. In the first workflow, first, the customers having a  
good payment history are initially selected (WS2,WS3), and then, the  
remaining customers whose credit history is below some threshold are  
filtered out (through WS1). The C2 workflow performs the same tasks in  
a reverse order. The above linear workflows may have different  
performance; if WS3 filters out more data than WS1, then it will be  
more beneficial to invoke WS3 before WS1 in order for the subsequent  
web services in the workflow to process less data.

It would be very useful to know if there exist similar scientific  
workflow examples (where users have many options for ordering the  
workflow tasks but cannot decide which task ordering to use, while the  
workflow performance depends on the workflow task invocation order)  
and if you are interested in extending swift with optimization  
algorithms for such workflows.

I am asking because i have recently developed an optimization  
algorithm for this problem and i would like to test its performance in  
a real-world workflow management system with real-world workflows.

P.S.: references to publications or any other information dealing with  
scientific workflows of the above rationale will be extremely useful.

Thank you very much for your time