From chad at uchicago.edu  Thu Mar  1 14:08:43 2007
From: chad at uchicago.edu (Chad Glendenin)
Date: Thu, 1 Mar 2007 14:08:43 -0600
Subject: [Swift-user] Command-line arg no longer working
Message-ID: <CCAD0085-EA29-4788-B05E-67FF969510E6@uchicago.edu>

Until today, I was running VDL2-rc3. Now I'm trying to run Swift  
snapshot 070228.

With rc3, this works:

$ vdlc minmax.dtm
$ vdlrun minmax.kml -list=filelist.txt

With Swift-070228, this fails:

$ vdlc minmax.dtm
$ swift minmax.kml -list=filelist.txt
Swift V 0.0405
RunID: rgxg81njt0fe0
Execution failed:
         Missing command line argument: list

I'm trying to use the csv_mapper, like this:

filelist_t dataset[] <csv_mapper; file=@arg("list")>;

Is there something I can do to fix this?

Thanks,
ccg



From hategan at mcs.anl.gov  Thu Mar  1 14:19:54 2007
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 01 Mar 2007 14:19:54 -0600
Subject: [Swift-user] Command-line arg no longer working
In-Reply-To: <CCAD0085-EA29-4788-B05E-67FF969510E6@uchicago.edu>
References: <CCAD0085-EA29-4788-B05E-67FF969510E6@uchicago.edu>
Message-ID: <1172780394.28884.6.camel@blabla.mcs.anl.gov>

Ooops. Sorry. Fixed in SVN.

Mihael

On Thu, 2007-03-01 at 14:08 -0600, Chad Glendenin wrote:
> Until today, I was running VDL2-rc3. Now I'm trying to run Swift  
> snapshot 070228.
> 
> With rc3, this works:
> 
> $ vdlc minmax.dtm
> $ vdlrun minmax.kml -list=filelist.txt
> 
> With Swift-070228, this fails:
> 
> $ vdlc minmax.dtm
> $ swift minmax.kml -list=filelist.txt
> Swift V 0.0405
> RunID: rgxg81njt0fe0
> Execution failed:
>          Missing command line argument: list
> 
> I'm trying to use the csv_mapper, like this:
> 
> filelist_t dataset[] <csv_mapper; file=@arg("list")>;
> 
> Is there something I can do to fix this?
> 
> Thanks,
> ccg
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 



From chad at uchicago.edu  Fri Mar  2 15:26:25 2007
From: chad at uchicago.edu (Chad Glendenin)
Date: Fri, 2 Mar 2007 15:26:25 -0600
Subject: [Swift-user] New Java exception when moving from rc3
Message-ID: <9B4531C0-72F7-461A-80CB-B7D050A6734A@uchicago.edu>

I'm still trying to run a workflow that worked with VDL rc3. With  
Swift snapshot 070301, the command-line argument works again, but now  
I get a Java exception:

$ swift minmax.kml -list=filelist.txt
Swift V 0.0405
RunID: flkjse5yhp1c0
Execution failed:
         java.lang.NumberFormatException: null

Here's the code I'm trying to run.

//////
type file_t {};
type filelist_t {
     file_t file;
}

(file_t output, file_t error) minmax (file_t input) {
     app {
         wrapper @input stdout=@output stderr=@error;
     }
}

//filelist_t dataset[] <csv_mapper; file="filelist.txt">;
filelist_t dataset[] <csv_mapper; file=@arg("list")>;
foreach f in dataset {
     file_t out <regexp_mapper; source=@f.file, match="([^/]+).tar",  
transform="\1.out">;
     file_t err <regexp_mapper; source=@f.file, match="([^/]+).tar",  
transform="\1.err">;
     (out, err) = minmax(f.file);
}
//////

Here's my input file, 'filelist.txt':

file
/scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000340.tar
/scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000341.tar

Any idea what's causing the exception?

By the way, the only reason I'm trying to upgrade is so that I can  
get disk-space management to work ("<profile namespace="SWIFT"  
key="storagesize">40000000000</profile>" in sites.xml). My  
understanding is that if I use VDL rc3, it could overrun the  
available scratch space and crash: I have 13.3 TB of data to process,  
but the UC/ANL TG site only has 1.1 TB of scratch space available. So  
if there's a way to enable scratch-space management in rc3, that  
should be good enough for now. (I'm a Swift user, not a developer, so  
I'd rather not be on the bleeding edge if I don't have to.)

Thanks,
ccg



From benc at hawaga.org.uk  Fri Mar  2 18:15:34 2007
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 3 Mar 2007 00:15:34 +0000 (GMT)
Subject: [Swift-user] swift 0.1
Message-ID: <Pine.LNX.4.64.0703030005350.2683@dildano.hawaga.org.uk>


I just put up v0.1 of Swift.

This is a small development release intended for users who prefer to use a 
released version rather than follow the nightly builds.

You can download from:
http://www.ci.uchicago.edu/swift/packages/vdsk-0.1.tar.gz

There's a changes file available at:

http://www.ci.uchicago.edu/trac/swift/browser/tags/v0.1rc1/CHANGES.txt?format=raw

Please send feedback to the swift-user or swift-devel at ci.uchicago.edu 
lists.

(note that if you are a nightly-build follower, this code is approximately 
equivalent to a build from a few days ago - bugfixes made in the past day 
or so are not present in v0.1)

--


From hategan at mcs.anl.gov  Fri Mar  2 18:31:59 2007
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 02 Mar 2007 18:31:59 -0600
Subject: [Swift-user] New Java exception when moving from rc3
In-Reply-To: <9B4531C0-72F7-461A-80CB-B7D050A6734A@uchicago.edu>
References: <9B4531C0-72F7-461A-80CB-B7D050A6734A@uchicago.edu>
Message-ID: <1172881919.16258.0.camel@blabla.mcs.anl.gov>

Chad,

Can you post the log file?

Mihael

On Fri, 2007-03-02 at 15:26 -0600, Chad Glendenin wrote:
> I'm still trying to run a workflow that worked with VDL rc3. With  
> Swift snapshot 070301, the command-line argument works again, but now  
> I get a Java exception:
> 
> $ swift minmax.kml -list=filelist.txt
> Swift V 0.0405
> RunID: flkjse5yhp1c0
> Execution failed:
>          java.lang.NumberFormatException: null
> 
> Here's the code I'm trying to run.
> 
> //////
> type file_t {};
> type filelist_t {
>      file_t file;
> }
> 
> (file_t output, file_t error) minmax (file_t input) {
>      app {
>          wrapper @input stdout=@output stderr=@error;
>      }
> }
> 
> //filelist_t dataset[] <csv_mapper; file="filelist.txt">;
> filelist_t dataset[] <csv_mapper; file=@arg("list")>;
> foreach f in dataset {
>      file_t out <regexp_mapper; source=@f.file, match="([^/]+).tar",  
> transform="\1.out">;
>      file_t err <regexp_mapper; source=@f.file, match="([^/]+).tar",  
> transform="\1.err">;
>      (out, err) = minmax(f.file);
> }
> //////
> 
> Here's my input file, 'filelist.txt':
> 
> file
> /scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000340.tar
> /scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000341.tar
> 
> Any idea what's causing the exception?
> 
> By the way, the only reason I'm trying to upgrade is so that I can  
> get disk-space management to work ("<profile namespace="SWIFT"  
> key="storagesize">40000000000</profile>" in sites.xml). My  
> understanding is that if I use VDL rc3, it could overrun the  
> available scratch space and crash: I have 13.3 TB of data to process,  
> but the UC/ANL TG site only has 1.1 TB of scratch space available. So  
> if there's a way to enable scratch-space management in rc3, that  
> should be good enough for now. (I'm a Swift user, not a developer, so  
> I'd rather not be on the bleeding edge if I don't have to.)
> 
> Thanks,
> ccg
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 



From chad at uchicago.edu  Fri Mar  2 18:39:16 2007
From: chad at uchicago.edu (Chad Glendenin)
Date: Fri, 2 Mar 2007 18:39:16 -0600
Subject: [Swift-user] New Java exception when moving from rc3
In-Reply-To: <1172881919.16258.0.camel@blabla.mcs.anl.gov>
References: <9B4531C0-72F7-461A-80CB-B7D050A6734A@uchicago.edu>
	<1172881919.16258.0.camel@blabla.mcs.anl.gov>
Message-ID: <45CDB132-66C9-4BF6-8E52-4E21B697961F@uchicago.edu>

$ cat swift.log
2007-03-02 14:25:07,474 DEBUG Loader Loader started

$ cat minmax-flkjse5yhp1c0.log
2007-03-02 14:25:09,645 INFO  unknown Using sites file: /scratch20/ 
chad/src/vdsk-070301/bin/../etc/sites.xml
2007-03-02 14:25:09,646 INFO  unknown Using tc.data: /scratch20/chad/ 
src/vdsk-070301/bin/../etc/tc.data

ccg

On Mar 2, 2007, at 6:31 PM, Mihael Hategan wrote:

> Chad,
>
> Can you post the log file?
>
> Mihael
>
> On Fri, 2007-03-02 at 15:26 -0600, Chad Glendenin wrote:
>> I'm still trying to run a workflow that worked with VDL rc3. With
>> Swift snapshot 070301, the command-line argument works again, but now
>> I get a Java exception:
>>
>> $ swift minmax.kml -list=filelist.txt
>> Swift V 0.0405
>> RunID: flkjse5yhp1c0
>> Execution failed:
>>          java.lang.NumberFormatException: null
>>
>> Here's the code I'm trying to run.
>>
>> //////
>> type file_t {};
>> type filelist_t {
>>      file_t file;
>> }
>>
>> (file_t output, file_t error) minmax (file_t input) {
>>      app {
>>          wrapper @input stdout=@output stderr=@error;
>>      }
>> }
>>
>> //filelist_t dataset[] <csv_mapper; file="filelist.txt">;
>> filelist_t dataset[] <csv_mapper; file=@arg("list")>;
>> foreach f in dataset {
>>      file_t out <regexp_mapper; source=@f.file, match="([^/]+).tar",
>> transform="\1.out">;
>>      file_t err <regexp_mapper; source=@f.file, match="([^/]+).tar",
>> transform="\1.err">;
>>      (out, err) = minmax(f.file);
>> }
>> //////
>>
>> Here's my input file, 'filelist.txt':
>>
>> file
>> /scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000340.tar
>> /scratch20/chad/src/workflow/driventurb_3d_direct_plt_cnt_000341.tar
>>
>> Any idea what's causing the exception?
>>
>> By the way, the only reason I'm trying to upgrade is so that I can
>> get disk-space management to work ("<profile namespace="SWIFT"
>> key="storagesize">40000000000</profile>" in sites.xml). My
>> understanding is that if I use VDL rc3, it could overrun the
>> available scratch space and crash: I have 13.3 TB of data to process,
>> but the UC/ANL TG site only has 1.1 TB of scratch space available. So
>> if there's a way to enable scratch-space management in rc3, that
>> should be good enough for now. (I'm a Swift user, not a developer, so
>> I'd rather not be on the bleeding edge if I don't have to.)
>>
>> Thanks,
>> ccg
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>
>



From wilde at mcs.anl.gov  Sat Mar  3 17:01:14 2007
From: wilde at mcs.anl.gov (Mike Wilde)
Date: Sat, 03 Mar 2007 17:01:14 -0600
Subject: [Swift-user] Returning boolean from a procedure
Message-ID: <45E9FE3A.1080102@mcs.anl.gov>


I think this was discussed a while back on the list but I cant find it: Can a 
procedure return a scalar, like Boolean, so that in the following tutorial 
example (q20) we can show the if-condition value coming from a function rather 
than a constant?

I also cant find where the current language open issues are.  Ben, I though 
you said these were in a bugzila bug but cant find it.

Sorry for asking redundant questions...

- Mike

type messagefile {}

(messagefile t) greeting (string s) {
     app {
         echo s stdout=@filename(t);
     }
}

messagefile outfile <"hello20.txt">;

boolean morning = true;

if(morning) {
   outfile = greeting("good morning");
} else {
   outfile = greeting("good afternoon");
}



From benc at hawaga.org.uk  Sat Mar  3 19:18:12 2007
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 4 Mar 2007 01:18:12 +0000 (GMT)
Subject: [Swift-user] Returning boolean from a procedure
In-Reply-To: <45E9FE3A.1080102@mcs.anl.gov>
References: <45E9FE3A.1080102@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0703040113220.6810@dildano.hawaga.org.uk>



On Sat, 3 Mar 2007, Mike Wilde wrote:

> I think this was discussed a while back on the list but I cant find it: Can a
> procedure return a scalar, like Boolean, so that in the following tutorial
> example (q20) we can show the if-condition value coming from a function rather
> than a constant?

not that function and procedure are different in SwiftScript - a procedure 
ends up being a unix invocation, and can only be called in syntax like:

  a = procedure(foo)

rather than being embedded inside an expression, so you can't say 
  a = 3 + procedure()

A function *doesn't* get executed remotely (at present, its something 
happening in the bowels of karajan).

If we had scalar returns from procedures (which we don't), we could say

 b = procedure(foo)
 if(b) { ...} else {...}

but we don't have scalar returns.

I too would like a slightly better example of if() for the tutorial.

> I also cant find where the current language open issues are.  Ben, I though
> you said these were in a bugzila bug but cant find it.

There's a VDL language issues wiki page on the VDL dev wiki - it has some 
fairly obvious name that I can't remember in my transient state. Some of 
the more targeted ones (i.e. ones that think should be fixed in the next 
couple of releases) are in the bugzilla; the wiki page has more rambly 
chatty material that no one has complained about enough to have it made 
into a targetted deliverable.

-- 


From benc at hawaga.org.uk  Sat Mar  3 19:53:51 2007
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 4 Mar 2007 01:53:51 +0000 (GMT)
Subject: [Swift-user] Returning boolean from a procedure
In-Reply-To: <45E9FE3A.1080102@mcs.anl.gov>
References: <45E9FE3A.1080102@mcs.anl.gov>
Message-ID: <Pine.LNX.4.64.0703040152180.6810@dildano.hawaga.org.uk>



On Sat, 3 Mar 2007, Mike Wilde wrote:

> I also cant find where the current language open issues are.  Ben, I though
> you said these were in a bugzila bug but cant find it.

also, not *a* bug - one bug per issue. but only issues that are concrete 
enough in terms of my understanding of them and in terms of having an 
actual app user who wants them (unlike a bunch of the wiki issues which 
are more theoretical). use 'change columns' in the search results page to 
turn on the Milestone column which may be useful.

-- 


From wuming at iit.edu  Wed Mar  7 15:48:30 2007
From: wuming at iit.edu (Ming Wu)
Date: Wed, 07 Mar 2007 15:48:30 -0600
Subject: [Swift-user] A question about resource selection strategy in swift
In-Reply-To: <Pine.LNX.4.64.0703030005350.2683@dildano.hawaga.org.uk>
References: <Pine.LNX.4.64.0703030005350.2683@dildano.hawaga.org.uk>
Message-ID: <e0a6d61ff820.f820e0a6d61f@iit.edu>

Hi,

When I submit a workflow to an environment having multiple sites, how 
swift distributes the tasks in a workflow onto those multiple 
resources? In other words, what strategies does swift use to transform 
an abstract workflow to an executable  workflow? 

Thanks

Ming




From yongzh at cs.uchicago.edu  Thu Mar  8 09:12:50 2007
From: yongzh at cs.uchicago.edu (Yong Zhao)
Date: Thu, 8 Mar 2007 09:12:50 -0600 (CST)
Subject: [Swift-user] A question about resource selection strategy in swift
In-Reply-To: <e0a6d61ff820.f820e0a6d61f@iit.edu>
References: <Pine.LNX.4.64.0703030005350.2683@dildano.hawaga.org.uk>
	<e0a6d61ff820.f820e0a6d61f@iit.edu>
Message-ID: <Pine.LNX.4.58.0703080911350.18397@classes.cs.uchicago.edu>

currently it is based on some heuristics. each site gets a score based on
its responsiveness, and jobs are dispatched proportionally according to
the scores the sites get.

Yong.

On Wed, 7 Mar 2007, Ming Wu wrote:

> Hi,
>
> When I submit a workflow to an environment having multiple sites, how
> swift distributes the tasks in a workflow onto those multiple
> resources? In other words, what strategies does swift use to transform
> an abstract workflow to an executable  workflow?
>
> Thanks
>
> Ming
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>