From wozniak at mcs.anl.gov  Mon Mar  1 15:30:03 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 1 Mar 2010 15:30:03 -0600 (CST)
Subject: [Swift-user] two questions about Swift (fwd)
Message-ID: <alpine.DEB.2.00.1003011528500.11329@roberts-desktop.mcs.anl.gov>

Hello Yi
 	If you can check out the latest Swift from trunk I've added some 
features that might help you out here.  There's a new built-in function @java() 
that allows you to call into an existing Java library.  You can call into the 
Java Platform or into your CLASSPATH.
 	Here is one example:

(float result) sin(float x) {
   result = @java("java.lang.Math", "sin", x);
}

float x = 0.5;
float y = sin(x);

trace("sin", x, y);

Note that you currently have to assign the result of @java() to a variable.

On Tue, 16 Feb 2010, Michael Wilde wrote:

>>>> Yi, sorry - I missed these questions from you; just saw them now.
>>>> 
>>>> ----- "Yi Zhu" <yizhu at cs.uchicago.edu> wrote:
>>>> 
>>>>> Hi all
>>>>> 
>>>>> 2.  In Matlab, there are some library which support some Math 
>>>>> functions like "sin" "cos" "reshape" etc. Is there any similar 
>>>>> library available for swift. If not, is there anyway to import 
>>>>> external library? (e.g. from Java)

-- 
Justin M Wozniak


From zhaozhang at uchicago.edu  Mon Mar  1 18:00:16 2010
From: zhaozhang at uchicago.edu (Zhao Zhang)
Date: Mon, 01 Mar 2010 18:00:16 -0600
Subject: [Swift-user] swift returns error on pads
In-Reply-To: <1267382291.6631.2.camel@localhost>
References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin>	
	<4B8A1115.7040803@uchicago.edu>
	<1267341005.3531.0.camel@localhost>	
	<4B8A9C05.4040408@uchicago.edu> <1267382291.6631.2.camel@localhost>
Message-ID: <4B8C5510.1090100@uchicago.edu>

Hi, Mihael

I found the following info in 
~/.globus/scripts/PBS9120483452797393898.submit.stderr

Statement unlikely to be reached at 
/home/zzhang/.globus/coasters/cscript553089225428166467.pl line 593.
        (Maybe you meant system() when you said exec()?)

Is there something wrong with the perl version? I am looking this up 
because I ran a simple swift script,
but it hangs there saying
Progress:  Selecting site:935  Submitted:65

Then no jobs run through. I checked "qstat -u zzhang", and my jobs were 
running.The log file is at
http://www.ci.uchicago.edu/~zzhang/movie-20100301-1725-5rzz1qt8.log

Thanks.

zhao

Mihael Hategan wrote:
> On Sun, 2010-02-28 at 10:38 -0600, Zhao Zhang wrote:
>   
>> Hi, Mihael
>>
>> It worked. But it only worked if I set the "maxnodes" to "2", does that 
>> mean "2" is the only number of
>> compute nodes that I could use on pads?
>>     
>
> I don't know what the various limits are on PADS, and I don't know how
> to find out short of sending mail to support (qstat -Q doesn't say
> much). But 2 nodes looks suspiciously low.
>
>
>   


From hategan at mcs.anl.gov  Mon Mar  1 18:01:25 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 01 Mar 2010 18:01:25 -0600
Subject: [Swift-user] swift returns error on pads
In-Reply-To: <4B8C5510.1090100@uchicago.edu>
References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin>
	<4B8A1115.7040803@uchicago.edu> <1267341005.3531.0.camel@localhost>
	<4B8A9C05.4040408@uchicago.edu> <1267382291.6631.2.camel@localhost>
	<4B8C5510.1090100@uchicago.edu>
Message-ID: <1267488085.7019.0.camel@localhost>

On Mon, 2010-03-01 at 18:00 -0600, Zhao Zhang wrote:
> Hi, Mihael
> 
> I found the following info in 
> ~/.globus/scripts/PBS9120483452797393898.submit.stderr
> 
> Statement unlikely to be reached at 
> /home/zzhang/.globus/coasters/cscript553089225428166467.pl line 593.
>         (Maybe you meant system() when you said exec()?)
> 
> Is there something wrong with the perl version?

No. Ignore that.




From zhaozhang at uchicago.edu  Tue Mar  2 11:56:08 2010
From: zhaozhang at uchicago.edu (Zhao Zhang)
Date: Tue, 02 Mar 2010 11:56:08 -0600
Subject: [Swift-user] pbs failure on pads
Message-ID: <4B8D5138.3010807@uchicago.edu>

Hi,

I am having the following failure right now on pads using coaster, it 
failed occasionally but unexpected.
I am not sure what the following info means, could some one point out? 
Thanks

[zzhang at login2 final]$ cat pbs.xml
<config>

  <pool handle="pbs">
    <execution provider="coaster" url="none" jobManager="local:pbs"/>
    <profile namespace="globus" key="queue">extended</profile>

    <profile namespace="globus" key="maxtime">3600</profile>
    <profile namespace="globus" key="maxwalltime">00:40:00</profile>   
    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="maxnodes">8</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="karajan" key="jobThrottle">.63</profile>

    <gridftp  url="local://localhost" />
    <workdirectory >/home/zzhang/swiftwork</workdirectory>
  </pool>
</config>


[zzhang at login2 final]$ swift -tc.file tc -sites.file pbs.xml movie.swift
Swift svn swift-r3255 (swift modified locally) cog-r2723

RunID: 20100302-1151-1tu5u5ac
Progress:
Progress:
Progress:
Progress:  uninitialized:1
Progress:  Initializing:16325  Selecting site:58
Progress:  Selecting site:16382  Initializing site shared directory:1
Progress:  Selecting site:16319  Stage in:63  Submitting:1
Progress:  Selecting site:16319  Stage in:46  Submitting:2  Submitted:16
Progress:  Selecting site:16319  Stage in:13  Submitting:1  Submitted:50
Progress:  Selecting site:16319  Submitted:63  Active:1
Progress:  Selecting site:16319  Submitted:60  Active:4
Progress:  Selecting site:16319  Submitted:45  Active:18  Checking 
status:1  Finished successfully:3
Progress:  Selecting site:16319  Submitting:1  Submitted:39  Active:21  
Checking status:1  Stage out:2  Finished successfully:13
Progress:  Selecting site:16319  Stage in:3  Submitted:33  Active:24  
Stage out:3  Finished successfully:17
Progress:  Selecting site:16317  Stage in:3  Submitted:35  Active:18  
Checking status:1  Stage out:7  Finished successfully:29
Progress:  Selecting site:16318  Stage in:11  Submitted:29  Active:24  
Finished successfully:42
Progress:  Selecting site:16318  Stage in:4  Submitted:34  Active:22  
Checking status:2  Stage out:2  Finished successfully:47
Progress:  Selecting site:16316  Stage in:6  Submitted:28  Active:23  
Checking status:1  Stage out:6  Finished successfully:55
Worker task failed: 0302-521133-000002 Block task ended prematurely
----------------------------------------
Begin PBS Prologue Tue Mar  2 11:52:38 CST 2010
Job ID:         6870.svc.pads.ci.uchicago.edu
Username:       zzhang
Group:          ci-users
Nodes:          
c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
End PBS Prologue Tue Mar  2 11:52:38 CST 2010
----------------------------------------
----------------------------------------
Begin PBS Epilogue Tue Mar  2 11:52:41 CST 2010
Job ID:         6870.svc.pads.ci.uchicago.edu
Username:       zzhang
Group:          ci-users
Job Name:       null
Session:        7051
Limits:         nodes=4,walltime=00:59:00
Resources:      cput=00:00:00,mem=700kb,vmem=8400kb,walltime=00:00:02
Nodes:          
c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
End PBS Epilogue Tue Mar  2 11:52:41 CST 2010

Progress:  Selecting site:16316  Stage in:6  Submitted:27  Active:23  
Stage out:7  Finished successfully:64 Failed but can retry:1
Failed to transfer wrapper log from movie-20100302-1151-1tu5u5ac/info/3 
on pbs
Execution failed:
        Exception in transform:
Arguments: [training_set/mv_0002679.txt]
Host: pbs
Directory: movie-20100302-1151-1tu5u5ac/jobs/3/transform-35lvrioj
stderr.txt:

stdout.txt:

----

Caused by:
        Task failed: 0302-521133-000002 Block task ended prematurely



From wilde at mcs.anl.gov  Tue Mar  2 12:03:06 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 2 Mar 2010 12:03:06 -0600 (CST)
Subject: [Swift-user] pbs failure on pads
In-Reply-To: <4B8D5138.3010807@uchicago.edu>
Message-ID: <30529233.1038641267552986202.JavaMail.root@zimbra>

Zhao, I was just debugging a similar problem. Mine turned out to be caused by leaving on replication in swift.properties. Try  setting replication.enabled=false in swift.properties, and let us know if that solves it.

You should also set the maxwalltime value in tc.data to the expected runtime of the app, and maxtime in the sites.xml entry to some multiple of that for the coaster block.

- Mike

----- "Zhao Zhang" <zhaozhang at uchicago.edu> wrote:

> Hi,
> 
> I am having the following failure right now on pads using coaster, it
> 
> failed occasionally but unexpected.
> I am not sure what the following info means, could some one point out?
> 
> Thanks
> 
> [zzhang at login2 final]$ cat pbs.xml
> <config>
> 
>   <pool handle="pbs">
>     <execution provider="coaster" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="queue">extended</profile>
> 
>     <profile namespace="globus" key="maxtime">3600</profile>
>     <profile namespace="globus" key="maxwalltime">00:40:00</profile>  
> 
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="maxnodes">8</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="karajan" key="jobThrottle">.63</profile>
> 
>     <gridftp  url="local://localhost" />
>     <workdirectory >/home/zzhang/swiftwork</workdirectory>
>   </pool>
> </config>
> 
> 
> [zzhang at login2 final]$ swift -tc.file tc -sites.file pbs.xml
> movie.swift
> Swift svn swift-r3255 (swift modified locally) cog-r2723
> 
> RunID: 20100302-1151-1tu5u5ac
> Progress:
> Progress:
> Progress:
> Progress:  uninitialized:1
> Progress:  Initializing:16325  Selecting site:58
> Progress:  Selecting site:16382  Initializing site shared directory:1
> Progress:  Selecting site:16319  Stage in:63  Submitting:1
> Progress:  Selecting site:16319  Stage in:46  Submitting:2 
> Submitted:16
> Progress:  Selecting site:16319  Stage in:13  Submitting:1 
> Submitted:50
> Progress:  Selecting site:16319  Submitted:63  Active:1
> Progress:  Selecting site:16319  Submitted:60  Active:4
> Progress:  Selecting site:16319  Submitted:45  Active:18  Checking 
> status:1  Finished successfully:3
> Progress:  Selecting site:16319  Submitting:1  Submitted:39  Active:21
>  
> Checking status:1  Stage out:2  Finished successfully:13
> Progress:  Selecting site:16319  Stage in:3  Submitted:33  Active:24 
> 
> Stage out:3  Finished successfully:17
> Progress:  Selecting site:16317  Stage in:3  Submitted:35  Active:18 
> 
> Checking status:1  Stage out:7  Finished successfully:29
> Progress:  Selecting site:16318  Stage in:11  Submitted:29  Active:24 
> 
> Finished successfully:42
> Progress:  Selecting site:16318  Stage in:4  Submitted:34  Active:22 
> 
> Checking status:2  Stage out:2  Finished successfully:47
> Progress:  Selecting site:16316  Stage in:6  Submitted:28  Active:23 
> 
> Checking status:1  Stage out:6  Finished successfully:55
> Worker task failed: 0302-521133-000002 Block task ended prematurely
> ----------------------------------------
> Begin PBS Prologue Tue Mar  2 11:52:38 CST 2010
> Job ID:         6870.svc.pads.ci.uchicago.edu
> Username:       zzhang
> Group:          ci-users
> Nodes:          
> c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
> End PBS Prologue Tue Mar  2 11:52:38 CST 2010
> ----------------------------------------
> ----------------------------------------
> Begin PBS Epilogue Tue Mar  2 11:52:41 CST 2010
> Job ID:         6870.svc.pads.ci.uchicago.edu
> Username:       zzhang
> Group:          ci-users
> Job Name:       null
> Session:        7051
> Limits:         nodes=4,walltime=00:59:00
> Resources:      cput=00:00:00,mem=700kb,vmem=8400kb,walltime=00:00:02
> Nodes:          
> c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
> End PBS Epilogue Tue Mar  2 11:52:41 CST 2010
> 
> Progress:  Selecting site:16316  Stage in:6  Submitted:27  Active:23 
> 
> Stage out:7  Finished successfully:64 Failed but can retry:1
> Failed to transfer wrapper log from
> movie-20100302-1151-1tu5u5ac/info/3 
> on pbs
> Execution failed:
>         Exception in transform:
> Arguments: [training_set/mv_0002679.txt]
> Host: pbs
> Directory: movie-20100302-1151-1tu5u5ac/jobs/3/transform-35lvrioj
> stderr.txt:
> 
> stdout.txt:
> 
> ----
> 
> Caused by:
>         Task failed: 0302-521133-000002 Block task ended prematurely
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From zhaozhang at uchicago.edu  Tue Mar  2 14:39:59 2010
From: zhaozhang at uchicago.edu (Zhao Zhang)
Date: Tue, 02 Mar 2010 14:39:59 -0600
Subject: [Swift-user] pbs failure on pads
In-Reply-To: <30529233.1038641267552986202.JavaMail.root@zimbra>
References: <30529233.1038641267552986202.JavaMail.root@zimbra>
Message-ID: <4B8D779F.7070304@uchicago.edu>

Hi, Mike

First I tried setting "replication.enabled=false", the failure still 
pops up.
Then I tried to set "maxwalltime" in tc.data, it doesn't solve the 
problem either.

zhao

Michael Wilde wrote:
> Zhao, I was just debugging a similar problem. Mine turned out to be caused by leaving on replication in swift.properties. Try  setting replication.enabled=false in swift.properties, and let us know if that solves it.
>
> You should also set the maxwalltime value in tc.data to the expected runtime of the app, and maxtime in the sites.xml entry to some multiple of that for the coaster block.
>
> - Mike
>
> ----- "Zhao Zhang" <zhaozhang at uchicago.edu> wrote:
>
>   
>> Hi,
>>
>> I am having the following failure right now on pads using coaster, it
>>
>> failed occasionally but unexpected.
>> I am not sure what the following info means, could some one point out?
>>
>> Thanks
>>
>> [zzhang at login2 final]$ cat pbs.xml
>> <config>
>>
>>   <pool handle="pbs">
>>     <execution provider="coaster" url="none" jobManager="local:pbs"/>
>>     <profile namespace="globus" key="queue">extended</profile>
>>
>>     <profile namespace="globus" key="maxtime">3600</profile>
>>     <profile namespace="globus" key="maxwalltime">00:40:00</profile>  
>>
>>     <profile namespace="globus" key="workersPerNode">8</profile>
>>     <profile namespace="globus" key="maxnodes">8</profile>
>>     <profile namespace="karajan" key="initialScore">10000</profile>
>>     <profile namespace="karajan" key="jobThrottle">.63</profile>
>>
>>     <gridftp  url="local://localhost" />
>>     <workdirectory >/home/zzhang/swiftwork</workdirectory>
>>   </pool>
>> </config>
>>
>>
>> [zzhang at login2 final]$ swift -tc.file tc -sites.file pbs.xml
>> movie.swift
>> Swift svn swift-r3255 (swift modified locally) cog-r2723
>>
>> RunID: 20100302-1151-1tu5u5ac
>> Progress:
>> Progress:
>> Progress:
>> Progress:  uninitialized:1
>> Progress:  Initializing:16325  Selecting site:58
>> Progress:  Selecting site:16382  Initializing site shared directory:1
>> Progress:  Selecting site:16319  Stage in:63  Submitting:1
>> Progress:  Selecting site:16319  Stage in:46  Submitting:2 
>> Submitted:16
>> Progress:  Selecting site:16319  Stage in:13  Submitting:1 
>> Submitted:50
>> Progress:  Selecting site:16319  Submitted:63  Active:1
>> Progress:  Selecting site:16319  Submitted:60  Active:4
>> Progress:  Selecting site:16319  Submitted:45  Active:18  Checking 
>> status:1  Finished successfully:3
>> Progress:  Selecting site:16319  Submitting:1  Submitted:39  Active:21
>>  
>> Checking status:1  Stage out:2  Finished successfully:13
>> Progress:  Selecting site:16319  Stage in:3  Submitted:33  Active:24 
>>
>> Stage out:3  Finished successfully:17
>> Progress:  Selecting site:16317  Stage in:3  Submitted:35  Active:18 
>>
>> Checking status:1  Stage out:7  Finished successfully:29
>> Progress:  Selecting site:16318  Stage in:11  Submitted:29  Active:24 
>>
>> Finished successfully:42
>> Progress:  Selecting site:16318  Stage in:4  Submitted:34  Active:22 
>>
>> Checking status:2  Stage out:2  Finished successfully:47
>> Progress:  Selecting site:16316  Stage in:6  Submitted:28  Active:23 
>>
>> Checking status:1  Stage out:6  Finished successfully:55
>> Worker task failed: 0302-521133-000002 Block task ended prematurely
>> ----------------------------------------
>> Begin PBS Prologue Tue Mar  2 11:52:38 CST 2010
>> Job ID:         6870.svc.pads.ci.uchicago.edu
>> Username:       zzhang
>> Group:          ci-users
>> Nodes:          
>> c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
>> End PBS Prologue Tue Mar  2 11:52:38 CST 2010
>> ----------------------------------------
>> ----------------------------------------
>> Begin PBS Epilogue Tue Mar  2 11:52:41 CST 2010
>> Job ID:         6870.svc.pads.ci.uchicago.edu
>> Username:       zzhang
>> Group:          ci-users
>> Job Name:       null
>> Session:        7051
>> Limits:         nodes=4,walltime=00:59:00
>> Resources:      cput=00:00:00,mem=700kb,vmem=8400kb,walltime=00:00:02
>> Nodes:          
>> c05.pads.ci.uchicago.edu,c15.pads.ci.uchicago.edu,c42.pads.ci.uchicago.edu,c43.pads.ci.uchicago.edu
>> End PBS Epilogue Tue Mar  2 11:52:41 CST 2010
>>
>> Progress:  Selecting site:16316  Stage in:6  Submitted:27  Active:23 
>>
>> Stage out:7  Finished successfully:64 Failed but can retry:1
>> Failed to transfer wrapper log from
>> movie-20100302-1151-1tu5u5ac/info/3 
>> on pbs
>> Execution failed:
>>         Exception in transform:
>> Arguments: [training_set/mv_0002679.txt]
>> Host: pbs
>> Directory: movie-20100302-1151-1tu5u5ac/jobs/3/transform-35lvrioj
>> stderr.txt:
>>
>> stdout.txt:
>>
>> ----
>>
>> Caused by:
>>         Task failed: 0302-521133-000002 Block task ended prematurely
>>
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>     
>
>   


From wilde at mcs.anl.gov  Wed Mar  3 09:43:55 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 3 Mar 2010 09:43:55 -0600 (CST)
Subject: [Swift-user] Re: [Swift-devel] Problem with coaster workers
	shutting down early
In-Reply-To: <1696956.1058231267576273817.JavaMail.root@zimbra>
Message-ID: <31008313.1070161267631035903.JavaMail.root@zimbra>

This is fixed in CoG rev 2725.

The problem was causing all multi-node coaster blocks to fail to start.

- Mike

----- wilde at mcs.anl.gov wrote:

> Mihael, I dont yet have all the evidence for this issue collected nice
> and clean, but I want to send you what I have to start looking at
> this.
> 
> Ive been trying to recreate a problem that Zhao is encountering where
> he's trying to run >15,000 short (~ 1-second) jobs on PADS under
> coasters.
> 
> Basically, the worker jobs seem to be exiting for no reason that I can
> discern.
> 
> Ive re-created something that looks similar using this:
> 
> cd ~wilde/swift/lab
> swift -tc.file tc -sites.file pbscoast.xml cats.swift
> 
> Log is /home/wilde/swift/lab/cats-20100302-1751-8qy7m21c.log
> 
> Coaster worker logs are in ~wilde/globus.coasters
> 
> Seems to work OK when I request 1 node blocks
> With 2-node blocks, the workers seem to shutdown for no apparent
> reason, after about 2 seconds.
> 
> ...more details later when I get a chance.
> 
> - Mike
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From mmaiensc at uchicago.edu  Wed Mar  3 15:38:47 2010
From: mmaiensc at uchicago.edu (Mark Maienschein-Cline)
Date: Wed, 3 Mar 2010 15:38:47 -0600
Subject: [Swift-user] Errors when not explicitly mapping outputs
Message-ID: <5199F013-0F6E-495D-91AA-747E13FE2729@uchicago.edu>

Hi,
I'm getting an error when I don't map the output filenames of applications explicitly, instead allowing swift to come up with its own names., e.g.

file input<"test">;
file output;
output = program( input );

fails, citing the fact that _concurrent/output-... was not created. If I map output explicitly, it runs fine. Is there an extra setting I need to make sure is turned on for swift to map its own file names?

Thanks,
Mark

From wilde at mcs.anl.gov  Wed Mar  3 19:35:20 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Wed, 3 Mar 2010 19:35:20 -0600 (CST)
Subject: [Swift-user] Errors when not explicitly mapping outputs
In-Reply-To: <13693759.1101821267665849716.JavaMail.root@zimbra>
Message-ID: <11914697.1102021267666520951.JavaMail.root@zimbra>

Mark, it looks to me like the example below *should* work. One possibility is that your program is getting a multi-component pathname (or a filename with unexpected characters like "-") as the output name, is not prepared to handle it, and is thus failing. Can you check if that's the case, perhaps by adding some debug output to your program? You can also perhaps see in the swift .log file if it looks like your program is getting invoked.

There's no setting needed for the default concurrent_mapper to be used as the default for output datasets.  Try it with echo as the program, like this:

login1$ cat concecho.swift
type file;

app (file o) echo (string s) { echo @s stdout=@o; }

file output = echo("I am concurrent!");
login1$ 
login1$ rm -rf _concurrent/
login1$ swift concecho.swift
Swift svn swift-r3255 (swift modified locally) cog-r2723

RunID: 20100303-1932-e3lu2ez4
Progress:
Final status:  Finished successfully:1
login1$ ls _concurrent/
output-0ac30d79-2f99-4baf-839c-878d8400ca9f-
login1$ cat _concurrent/output-0ac30d79-2f99-4baf-839c-878d8400ca9f- 
I am concurrent!
login1$ 



- Mike


----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:

> Hi,
> I'm getting an error when I don't map the output filenames of
> applications explicitly, instead allowing swift to come up with its
> own names., e.g.
> 
> file input<"test">;
> file output;
> output = program( input );
> 
> fails, citing the fact that _concurrent/output-... was not created. If
> I map output explicitly, it runs fine. Is there an extra setting I
> need to make sure is turned on for swift to map its own file names?
> 
> Thanks,
> Mark_______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wilde at mcs.anl.gov  Fri Mar  5 18:26:10 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 5 Mar 2010 18:26:10 -0600 (CST)
Subject: [Swift-user] sites.xml entry for coasters on PADS
Message-ID: <16507162.1165181267835170637.JavaMail.root@zimbra>


Here is an example of the sites.xml element I used for running on the CI PADS PBS cluster. It can be used on pretty much any PBS system.

This for for running directly on the PBS headnode:

  <pool handle="pbs">
    <execution provider="coaster" url="none" jobmanager="local:pbs"/>
    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="maxTime">3500</profile>
    <profile namespace="globus" key="maxWallTime">00:02:00</profile>
    <profile namespace="globus" key="slots">4</profile>
    <profile namespace="globus" key="nodeGranularity">4</profile>
    <profile namespace="globus" key="maxNodes">32</profile>
    <profile namespace="karajan" key="jobThrottle">2.55</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <filesystem provider="local" url="none"/>
    <workdirectory>/home/wilde/swiftwork</workdirectory>
  </pool>

You need to adjust the workdir and times according to your needs.
And we can fiddle with the granularity etc.

Ive also run ssh'ing to PADS from elsewhere, since it doesnt have Globus GRAM, using this entry:

  <pool handle="pbs">
    <execution provider="coaster" url="login.pads.ci.uchicago.edu" jobmanager="ssh:pbs"/>

    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="slots">1</profile>
    <profile namespace="globus" key="nodeGranularity">2</profile>
    <profile namespace="globus" key="maxNodes">2</profile>
    <profile namespace="karajan" key="jobThrottle">.63</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>

    <filesystem provider="ssh" url="login.pads.ci.uchicago.edu"/>
    <workdirectory>/home/wilde/swiftwork</workdirectory>
  </pool>


In this case I used a /home/wilde/.ssh/auth.defaults file with something like this:

login.pads.ci.uchicago.edu.type=key
login.pads.ci.uchicago.edu.username=wilde
login.pads.ci.uchicago.edu.key=/home/wilde/.ssh/swift_rsa
login.pads.ci.uchicago.edu.passphrase=mypassphrasegoeshere

I will try to get this more well-documented in the Users Guide or SWFT wiki. For now, feel free to fire questions about it to the Swift-User list.

- Mike






-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From zhangzhao0718 at gmail.com  Sat Mar  6 16:08:29 2010
From: zhangzhao0718 at gmail.com (Zhao Zhang)
Date: Sat, 06 Mar 2010 16:08:29 -0600
Subject: [Swift-user] swift-plot-log error
Message-ID: <4B92D25D.7090605@gmail.com>

Hi,

I encountered an error when plotting the logs of my swift runs.
extract-coaster-timeline 
/home/zzhang/workplace/bigdata/final/movie-20100306-1206-ht6vcu74.log
make: execvp: extract-coaster-timeline: Permission denied
make: *** [extract-timeline] Error 127

Any idea about this? Thanks.

Best
zhao


From wilde at mcs.anl.gov  Sun Mar  7 13:57:40 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sun, 7 Mar 2010 13:57:40 -0600 (CST)
Subject: [Swift-user] swift-plot-log error
In-Reply-To: <17980413.1176911267991815571.JavaMail.root@zimbra>
Message-ID: <7392546.1176931267991860195.JavaMail.root@zimbra>

I think the problem is simply that several coaster plot tools didnt get set executable. For the moment, just do that manually, and let me know if it works; then I'll fix in svn.

- Mike

in the distribution dir libexec/log-processing:

  -rw-r--r-- 1 wilde ci-users   245 Mar  5 18:41 coaster-block-timeline.plot
  -rw-r--r-- 1 wilde ci-users   178 Mar  5 18:41 coaster-block-utilization-vs-count.plot
  -rw-r--r-- 1 wilde ci-users   160 Mar  5 18:41 coaster-block-utilization.plot
  -rw-r--r-- 1 wilde ci-users   147 Mar  5 18:41 coaster-qwait-count.plot
  -rw-r--r-- 1 wilde ci-users   163 Mar  5 18:41 coaster-qwait-size.plot
  -rw-r--r-- 1 wilde ci-users   143 Mar  5 18:41 coaster-qwait-wtime.plot



----- "Zhao Zhang" <zhangzhao0718 at gmail.com> wrote:

> Hi,
> 
> I encountered an error when plotting the logs of my swift runs.
> extract-coaster-timeline 
> /home/zzhang/workplace/bigdata/final/movie-20100306-1206-ht6vcu74.log
> make: execvp: extract-coaster-timeline: Permission denied
> make: *** [extract-timeline] Error 127
> 
> Any idea about this? Thanks.
> 
> Best
> zhao
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From zhaozhang at uchicago.edu  Sun Mar  7 14:11:49 2010
From: zhaozhang at uchicago.edu (Zhao Zhang)
Date: Sun, 07 Mar 2010 14:11:49 -0600
Subject: [Swift-user] swift-plot-log error
In-Reply-To: <7392546.1176931267991860195.JavaMail.root@zimbra>
References: <7392546.1176931267991860195.JavaMail.root@zimbra>
Message-ID: <4B940885.6090205@uchicago.edu>

Thanks, Mike. Allan also pointed me to this, and it works now.

zhao

wilde at mcs.anl.gov wrote:
> I think the problem is simply that several coaster plot tools didnt get set executable. For the moment, just do that manually, and let me know if it works; then I'll fix in svn.
>
> - Mike
>
> in the distribution dir libexec/log-processing:
>
>   -rw-r--r-- 1 wilde ci-users   245 Mar  5 18:41 coaster-block-timeline.plot
>   -rw-r--r-- 1 wilde ci-users   178 Mar  5 18:41 coaster-block-utilization-vs-count.plot
>   -rw-r--r-- 1 wilde ci-users   160 Mar  5 18:41 coaster-block-utilization.plot
>   -rw-r--r-- 1 wilde ci-users   147 Mar  5 18:41 coaster-qwait-count.plot
>   -rw-r--r-- 1 wilde ci-users   163 Mar  5 18:41 coaster-qwait-size.plot
>   -rw-r--r-- 1 wilde ci-users   143 Mar  5 18:41 coaster-qwait-wtime.plot
>
>
>
> ----- "Zhao Zhang" <zhangzhao0718 at gmail.com> wrote:
>
>   
>> Hi,
>>
>> I encountered an error when plotting the logs of my swift runs.
>> extract-coaster-timeline 
>> /home/zzhang/workplace/bigdata/final/movie-20100306-1206-ht6vcu74.log
>> make: execvp: extract-coaster-timeline: Permission denied
>> make: *** [extract-timeline] Error 127
>>
>> Any idea about this? Thanks.
>>
>> Best
>> zhao
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>>     
>
>   


From nbest at ci.uchicago.edu  Wed Mar 10 13:24:34 2010
From: nbest at ci.uchicago.edu (Neil Best)
Date: Wed, 10 Mar 2010 13:24:34 -0600
Subject: [Swift-user] coaster workflow hangs then dies
Message-ID: <D5D588E0-80E7-4906-A72D-A5565F4D684C@ci.uchicago.edu>

Please take a look at login.ci:~nbest/bigdata/files/ 
mcd12q1-20100310-1227-6gy82bq0.stdout and associated files.  There are  
some exceptions in middle of the .stdout abd then it appears to fail  
at the end:

Progress:  Selecting site:3519  Stage in:70  Stage out:26  Finished  
successfully:189
Progress:  Selecting site:3519  Stage in:69  Submitting:1  Stage out: 
26  Finished successfully:189
Progress:  Selecting site:3519  Stage in:67  Active:2  Checking status: 
1  Stage out:26  Finished successfully:189
Cleaning up...
Shutdown failed after 5 minutes. Forcefully shutting down
Progress:Progress:Shutting down service at https://192.5.86.6:51333
   Selecting site:3519  Selecting site:3519  Stage in:60  Stage in:61   
Submitting:3Got channel MetaChannel: 1626091176 -> null  Submitting:2
   Checking status:5  Stage out:28  Checking status:5  Stage out:28   
Finished successfully:189
   Finished successfully:189
Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:1   
Checking status:4  Stage out:28  Finished successfully:189
Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:2   
Checking status:4  Stage out:27  Finished successfully:189
Progress:  Selecting site:3519  Stage in:61  Submitting:3  Active:2   
Checking status:3  Stage out:27  Finished successfully:189
Progress:  Selecting site:3519  Stage in:65  Submitting:1  Checking  
status:3  Stage out:27  Finished successfully:189
Progress:  Selecting site:3519  Stage in:66  Checking status:3  Stage  
out:27  Finished successfully:189
Progress:  Selecting site:3519  Stage in:67  Checking status:2  Stage  
out:27  Finished successfully:189
Progress:  Selecting site:3519  Stage in:50  Submitting:13  Checking  
status:5  Stage out:28  Finished successfully:189
+ Done


Can anyone tell me what might be the cause of this?


From aespinosa at cs.uchicago.edu  Wed Mar 10 17:06:58 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 10 Mar 2010 17:06:58 -0600
Subject: [Swift-user] simple_mapper on simple data types.
Message-ID: <50b07b4b1003101506s74a37f12u8d923293e950d0c0@mail.gmail.com>

Hi,

I'm getting this response from swift:
/home/wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift
-dryrun -runid test runchange.swift

Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified locally)

RunID: test
Progress:
Execution failed:
	java.lang.IllegalStateException: mapper.existing() returned a path
..A2001001.h29v08.005.2009342154532 that it cannot subsequently map


Here's the script:
type Hdf4Eos;
type MapPng;
type Hdf5;

Hdf4Eos before <simple_mapper;prefix="MCD12Q1",pattern="*h29v08*",
    suffix=".hdf">;
trace(@filename(before));

Here's the file that it was trying to map (for input later on).
$ ls *.hdf
MCD12Q1.A2001001.h29v08.005.2009342154532.hdf

I haven't found a good reference for doing this as most of the
examples in tests/language-behavior uses structs or arrays together
with simple_mapper for output mapping

thanks,
-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Wed Mar 10 17:18:24 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 10 Mar 2010 17:18:24 -0600
Subject: [Swift-user] Re: simple_mapper on simple data types.
In-Reply-To: <50b07b4b1003101506s74a37f12u8d923293e950d0c0@mail.gmail.com>
References: <50b07b4b1003101506s74a37f12u8d923293e950d0c0@mail.gmail.com>
Message-ID: <50b07b4b1003101518l1c842896t4868c458baae2220@mail.gmail.com>

I did a quick hack of zero element arrays and it seemed to work

   Hdf4Eos before[] <filesys_mapper;location=getdir(by),
        prefix="MCD12Q1",pattern=getpattern(tile), suffix=".hdf">;
    trace(@filename(before[0]));

2010/3/10 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> Hi,
>
> I'm getting this response from swift:
> /home/wilde/swift/src/stable/cog/modules/swift/dist/swift-svn/bin/swift
> -dryrun -runid test runchange.swift
>
> Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified locally)
>
> RunID: test
> Progress:
> Execution failed:
> ? ? ? ?java.lang.IllegalStateException: mapper.existing() returned a path
> ..A2001001.h29v08.005.2009342154532 that it cannot subsequently map
>
>
> Here's the script:
> type Hdf4Eos;
> type MapPng;
> type Hdf5;
>
> Hdf4Eos before <simple_mapper;prefix="MCD12Q1",pattern="*h29v08*",
> ? ?suffix=".hdf">;
> trace(@filename(before));
>
> Here's the file that it was trying to map (for input later on).
> $ ls *.hdf
> MCD12Q1.A2001001.h29v08.005.2009342154532.hdf
>
> I haven't found a good reference for doing this as most of the
> examples in tests/language-behavior uses structs or arrays together
> with simple_mapper for output mapping
>
> thanks,
> -Allan


From hategan at mcs.anl.gov  Wed Mar 10 18:07:50 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 10 Mar 2010 18:07:50 -0600
Subject: [Swift-user] coaster workflow hangs then dies
In-Reply-To: <D5D588E0-80E7-4906-A72D-A5565F4D684C@ci.uchicago.edu>
References: <D5D588E0-80E7-4906-A72D-A5565F4D684C@ci.uchicago.edu>
Message-ID: <1268266070.11181.0.camel@localhost>

Did you at some point during the run press CTRL+C or otherwise interrupt
the swift process?

On Wed, 2010-03-10 at 13:24 -0600, Neil Best wrote:
> Please take a look at login.ci:~nbest/bigdata/files/ 
> mcd12q1-20100310-1227-6gy82bq0.stdout and associated files.  There are  
> some exceptions in middle of the .stdout abd then it appears to fail  
> at the end:
> 
> Progress:  Selecting site:3519  Stage in:70  Stage out:26  Finished  
> successfully:189
> Progress:  Selecting site:3519  Stage in:69  Submitting:1  Stage out: 
> 26  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:67  Active:2  Checking status: 
> 1  Stage out:26  Finished successfully:189
> Cleaning up...
> Shutdown failed after 5 minutes. Forcefully shutting down
> Progress:Progress:Shutting down service at https://192.5.86.6:51333
>    Selecting site:3519  Selecting site:3519  Stage in:60  Stage in:61   
> Submitting:3Got channel MetaChannel: 1626091176 -> null  Submitting:2
>    Checking status:5  Stage out:28  Checking status:5  Stage out:28   
> Finished successfully:189
>    Finished successfully:189
> Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:1   
> Checking status:4  Stage out:28  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:2   
> Checking status:4  Stage out:27  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:61  Submitting:3  Active:2   
> Checking status:3  Stage out:27  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:65  Submitting:1  Checking  
> status:3  Stage out:27  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:66  Checking status:3  Stage  
> out:27  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:67  Checking status:2  Stage  
> out:27  Finished successfully:189
> Progress:  Selecting site:3519  Stage in:50  Submitting:13  Checking  
> status:5  Stage out:28  Finished successfully:189
> + Done
> 
> 
> Can anyone tell me what might be the cause of this?
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user



From hategan at mcs.anl.gov  Wed Mar 10 18:11:30 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 10 Mar 2010 18:11:30 -0600
Subject: [Swift-user] coaster workflow hangs then dies
In-Reply-To: <1268266070.11181.0.camel@localhost>
References: <D5D588E0-80E7-4906-A72D-A5565F4D684C@ci.uchicago.edu>
	<1268266070.11181.0.camel@localhost>
Message-ID: <1268266290.11181.2.camel@localhost>

Nevermind that. You didn't.

I think what happens is that during file transfers the service is
wrongly not considered to be active. It eventually reaches its maximum
allowed idle time and then it shuts down.

I'll try to put in a fix for this.

Thanks for uncovering this.

Mihael

On Wed, 2010-03-10 at 18:07 -0600, Mihael Hategan wrote:
> Did you at some point during the run press CTRL+C or otherwise interrupt
> the swift process?
> 
> On Wed, 2010-03-10 at 13:24 -0600, Neil Best wrote:
> > Please take a look at login.ci:~nbest/bigdata/files/ 
> > mcd12q1-20100310-1227-6gy82bq0.stdout and associated files.  There are  
> > some exceptions in middle of the .stdout abd then it appears to fail  
> > at the end:
> > 
> > Progress:  Selecting site:3519  Stage in:70  Stage out:26  Finished  
> > successfully:189
> > Progress:  Selecting site:3519  Stage in:69  Submitting:1  Stage out: 
> > 26  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:67  Active:2  Checking status: 
> > 1  Stage out:26  Finished successfully:189
> > Cleaning up...
> > Shutdown failed after 5 minutes. Forcefully shutting down
> > Progress:Progress:Shutting down service at https://192.5.86.6:51333
> >    Selecting site:3519  Selecting site:3519  Stage in:60  Stage in:61   
> > Submitting:3Got channel MetaChannel: 1626091176 -> null  Submitting:2
> >    Checking status:5  Stage out:28  Checking status:5  Stage out:28   
> > Finished successfully:189
> >    Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:1   
> > Checking status:4  Stage out:28  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:61  Submitting:2  Active:2   
> > Checking status:4  Stage out:27  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:61  Submitting:3  Active:2   
> > Checking status:3  Stage out:27  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:65  Submitting:1  Checking  
> > status:3  Stage out:27  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:66  Checking status:3  Stage  
> > out:27  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:67  Checking status:2  Stage  
> > out:27  Finished successfully:189
> > Progress:  Selecting site:3519  Stage in:50  Submitting:13  Checking  
> > status:5  Stage out:28  Finished successfully:189
> > + Done
> > 
> > 
> > Can anyone tell me what might be the cause of this?
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user



From mmaiensc at uchicago.edu  Fri Mar 12 15:53:17 2010
From: mmaiensc at uchicago.edu (Mark Maienschein-Cline)
Date: Fri, 12 Mar 2010 15:53:17 -0600
Subject: [Swift-user] Running swift with the sge provider
Message-ID: <D22B488D-BB1E-431C-BA3A-968322D0655B@uchicago.edu>

Hi,
I've tried running swift with the sge provider, and I get the following error:

Swift svn swift-r3255 (swift modified locally) cog-r2723 (cog modified locally)

RunID: 20100312-1646-8zx0vx6f
Progress:
Progress:  Submitted:1
Failed to transfer wrapper log from pipeline2-20100312-1646-8zx0vx6f/info/r on sge
Execution failed:
	Exception in comppeaks:
Arguments: [export3/chip-seq/analyses/mmc_analyses/mmc-irf4-D3-2_q1.1pct.xls export3/chip-seq/analyses/mmc_analyses/mmc-irf4-D3-3_q1.1pct.xls, 200, export3/chip-seq/analyses/mmc_analyses/mmc-irf4-D3-overlap.xls]
Host: sge
Directory: pipeline2-20100312-1646-8zx0vx6f/jobs/r/comppeaks-rkd2i0pj
stderr.txt: 

stdout.txt: 

----

Caused by:
	Could not submit job
Caused by:
	Could not start coaster service
Caused by:
	java.lang.IllegalArgumentException: Address type not supported
Cleaning up...
 Done

I'm running a stable version of swift, download from PADS on March 3. I've attached the swift script, tc file, and sites file I used, and the log file. Is there something else my cluster needs to run swift on the nodes (my cluster is sisboombah.uchicago.edu)? This script does work locally.
Thanks,
Mark


-------------- next part --------------
A non-text attachment was scrubbed...
Name: pipeline2.swift
Type: application/octet-stream
Size: 1695 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100312/1adb76f2/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tc
Type: application/octet-stream
Size: 1767 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100312/1adb76f2/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sge.xml
Type: application/xml
Size: 1959 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100312/1adb76f2/attachment.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pipeline2-20100312-1646-8zx0vx6f.log
Type: application/octet-stream
Size: 64406 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100312/1adb76f2/attachment-0002.obj>
-------------- next part --------------



From hategan at mcs.anl.gov  Fri Mar 12 15:59:00 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 12 Mar 2010 15:59:00 -0600
Subject: [Swift-user] Running swift with the sge provider
In-Reply-To: <D22B488D-BB1E-431C-BA3A-968322D0655B@uchicago.edu>
References: <D22B488D-BB1E-431C-BA3A-968322D0655B@uchicago.edu>
Message-ID: <1268431140.26041.0.camel@localhost>

I'd try url="localhost" instead of url="none" in sge.xml

On Fri, 2010-03-12 at 15:53 -0600, Mark Maienschein-Cline wrote:

> Caused by:
> 	Could not submit job
> Caused by:
> 	Could not start coaster service
> Caused by:
> 	java.lang.IllegalArgumentException: Address type not supported
> Cleaning up...
>  Done




From hategan at mcs.anl.gov  Fri Mar 12 16:09:53 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 12 Mar 2010 16:09:53 -0600
Subject: [Swift-user] Running swift with the sge provider
In-Reply-To: <AC933AE5-5F06-462F-A3A3-4FF7EB85B3F3@uchicago.edu>
References: <D22B488D-BB1E-431C-BA3A-968322D0655B@uchicago.edu>
	<1268431140.26041.0.camel@localhost>
	<AC933AE5-5F06-462F-A3A3-4FF7EB85B3F3@uchicago.edu>
Message-ID: <1268431793.26171.2.camel@localhost>

Ooops. Right. The GCJ conspiracy is getting more subtle.

Try a proper JVM/JDK. You seem to be using GCJ which, well, is at least
trying very hard.

On Fri, 2010-03-12 at 16:02 -0600, Mark Maienschein-Cline wrote:
> Got the same error with that change.
> 
> 
> On Mar 12, 2010, at 3:59 PM, Mihael Hategan wrote:
> 
> > I'd try url="localhost" instead of url="none" in sge.xml
> > 
> > On Fri, 2010-03-12 at 15:53 -0600, Mark Maienschein-Cline wrote:
> > 
> >> Caused by:
> >> 	Could not submit job
> >> Caused by:
> >> 	Could not start coaster service
> >> Caused by:
> >> 	java.lang.IllegalArgumentException: Address type not supported
> >> Cleaning up...
> >> Done
> > 
> > 
> 



From hategan at mcs.anl.gov  Fri Mar 12 16:35:11 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 12 Mar 2010 16:35:11 -0600
Subject: [Swift-user] Running swift with the sge provider
In-Reply-To: <3184BD0E-56AE-4B3E-BE8C-35BAEDFF367E@uchicago.edu>
References: <D22B488D-BB1E-431C-BA3A-968322D0655B@uchicago.edu>
	<1268431140.26041.0.camel@localhost>
	<AC933AE5-5F06-462F-A3A3-4FF7EB85B3F3@uchicago.edu>
	<1268431793.26171.2.camel@localhost>
	<3184BD0E-56AE-4B3E-BE8C-35BAEDFF367E@uchicago.edu>
Message-ID: <1268433311.26352.3.camel@localhost>

They aren't only different compilers. GCJ also provides its own
re-implementation of the java core libraries. Unfortunately GCJ doesn't
quite work.

I suspect the sun JDK is already installed on the cluster. Try playing
with softenv or ask an admin.

On Fri, 2010-03-12 at 16:20 -0600, Mark Maienschein-Cline wrote:
> I sort of see... these are different compilers? Does this mean we will have to change the java compiler on the cluster?
> 
> Thanks,
> Mark
> 
> On Mar 12, 2010, at 4:09 PM, Mihael Hategan wrote:
> 
> > Ooops. Right. The GCJ conspiracy is getting more subtle.
> > 
> > Try a proper JVM/JDK. You seem to be using GCJ which, well, is at least
> > trying very hard.
> > 
> > On Fri, 2010-03-12 at 16:02 -0600, Mark Maienschein-Cline wrote:
> >> Got the same error with that change.
> >> 
> >> 
> >> On Mar 12, 2010, at 3:59 PM, Mihael Hategan wrote:
> >> 
> >>> I'd try url="localhost" instead of url="none" in sge.xml
> >>> 
> >>> On Fri, 2010-03-12 at 15:53 -0600, Mark Maienschein-Cline wrote:
> >>> 
> >>>> Caused by:
> >>>> 	Could not submit job
> >>>> Caused by:
> >>>> 	Could not start coaster service
> >>>> Caused by:
> >>>> 	java.lang.IllegalArgumentException: Address type not supported
> >>>> Cleaning up...
> >>>> Done
> >>> 
> >>> 
> >> 
> > 
> 



From marcin at galton.uchicago.edu  Sat Mar 13 10:04:07 2010
From: marcin at galton.uchicago.edu (Marcin Hitczenko)
Date: Sat, 13 Mar 2010 10:04:07 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
Message-ID: <36454.207.181.247.181.1268496247.squirrel@galton.uchicago.edu>

Hi,

I am trying to run swift scripts on fusion and I am encountering an error
I have never had before (The scripts I am running have worked before). It
seems it is having trouble submitting job because it "failed to transfer
wrapper log". I am including my swift script, tc.data, sites.xml and the
output file of when I ran it.

I am not sure if I need to change anything? Like I said, I am sure the
same script worked a month or so ago.

Thanks for your help.

Best,

Marcin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: swift.out
Type: application/octet-stream
Size: 3348 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100313/4e5c36a3/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getcoeffs.swift
Type: application/octet-stream
Size: 3430 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100313/4e5c36a3/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sites.xml
Type: text/xml
Size: 374 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100313/4e5c36a3/attachment.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tc.data
Type: application/octet-stream
Size: 1069 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100313/4e5c36a3/attachment-0002.obj>

From wilde at mcs.anl.gov  Sat Mar 13 10:24:56 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 13 Mar 2010 10:24:56 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
In-Reply-To: <36454.207.181.247.181.1268496247.squirrel@galton.uchicago.edu>
Message-ID: <573276.1271268497496362.JavaMail.root@zimbra>

Marcin, can you also post the .log file from this run? (it will be named getcoefs*.log where * is a long unique id including the date)

- Mike

----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:

> Hi,
> 
> I am trying to run swift scripts on fusion and I am encountering an
> error
> I have never had before (The scripts I am running have worked before).
> It
> seems it is having trouble submitting job because it "failed to
> transfer
> wrapper log". I am including my swift script, tc.data, sites.xml and
> the
> output file of when I ran it.
> 
> I am not sure if I need to change anything? Like I said, I am sure
> the
> same script worked a month or so ago.
> 
> Thanks for your help.
> 
> Best,
> 
> Marcin
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 13 10:36:01 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 13 Mar 2010 10:36:01 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
In-Reply-To: <573276.1271268497496362.JavaMail.root@zimbra>
Message-ID: <22222730.1401268498161973.JavaMail.root@zimbra>

Marcin, I forgot to point out: "failed to transfer wrapper log" is just a catch-all error message which means "something went wrong with an app() job that Swift ran, and the job did not return the expected log file that comes from the wrapper script under which Swift runs the job.  We need to improve the text of this message.

Also, if you can, always run swift with standard output and error redirected into a file, and send that file as well when you report a problem.

Thanks,

Mike

----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> Marcin, can you also post the .log file from this run? (it will be
> named getcoefs*.log where * is a long unique id including the date)
> 
> - Mike
> 
> ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> 
> > Hi,
> > 
> > I am trying to run swift scripts on fusion and I am encountering an
> > error
> > I have never had before (The scripts I am running have worked
> before).
> > It
> > seems it is having trouble submitting job because it "failed to
> > transfer
> > wrapper log". I am including my swift script, tc.data, sites.xml
> and
> > the
> > output file of when I ran it.
> > 
> > I am not sure if I need to change anything? Like I said, I am sure
> > the
> > same script worked a month or so ago.
> > 
> > Thanks for your help.
> > 
> > Best,
> > 
> > Marcin
> > 
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> 
> -- 
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 13 11:04:42 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 13 Mar 2010 11:04:42 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
In-Reply-To: <46026.207.181.247.181.1268498468.squirrel@galton.uchicago.edu>
Message-ID: <9553378.1591268499882482.JavaMail.root@zimbra>

Marcin,

I see two problems, the second likely being the result of the first:

In your output file: [ERROR] Parsing profiles on line 21 Illegal character ':'at position 22 :Illegal character ':'
is referring to your tc.data file. I think your Globus MaxWallTime profile entry got moved to a separate line, instead of being separated by tabs as the last column of the previous line.

I suspect that may have caused jobs to get submitted to PBS with defaults that were invalid for the default queue that your jobs are going into, thus causing the second error: Cannot submit job: Could not submit job (qsub reported an exit code of 1). no error output

So fix tc.data, and see if this fixes the problem.

- Mike

----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:

> Hi Mike,
> 
> Thanks for your response. I am attaching the .log file. Also, the
> swift.out file I included in the original email has the output of my
> run.
> I am including it again.
> 
> Best,
> 
> Marcin
> 
> > Marcin, I forgot to point out: "failed to transfer wrapper log" is
> just a
> > catch-all error message which means "something went wrong with an
> app()
> > job that Swift ran, and the job did not return the expected log file
> that
> > comes from the wrapper script under which Swift runs the job.  We
> need to
> > improve the text of this message.
> >
> > Also, if you can, always run swift with standard output and error
> > redirected into a file, and send that file as well when you report
> a
> > problem.
> >
> > Thanks,
> >
> > Mike
> >
> > ----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> >
> >> Marcin, can you also post the .log file from this run? (it will be
> >> named getcoefs*.log where * is a long unique id including the
> date)
> >>
> >> - Mike
> >>
> >> ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying to run swift scripts on fusion and I am encountering
> an
> >> > error
> >> > I have never had before (The scripts I am running have worked
> >> before).
> >> > It
> >> > seems it is having trouble submitting job because it "failed to
> >> > transfer
> >> > wrapper log". I am including my swift script, tc.data, sites.xml
> >> and
> >> > the
> >> > output file of when I ran it.
> >> >
> >> > I am not sure if I need to change anything? Like I said, I am
> sure
> >> > the
> >> > same script worked a month or so ago.
> >> >
> >> > Thanks for your help.
> >> >
> >> > Best,
> >> >
> >> > Marcin
> >> >
> >> > _______________________________________________
> >> > Swift-user mailing list
> >> > Swift-user at ci.uchicago.edu
> >> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >>
> >> --
> >> Michael Wilde
> >> Computation Institute, University of Chicago
> >> Mathematics and Computer Science Division
> >> Argonne National Laboratory
> >>
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 13 12:08:55 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 13 Mar 2010 12:08:55 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
In-Reply-To: <41383.207.181.247.181.1268501339.squirrel@galton.uchicago.edu>
Message-ID: <8219211.2381268503735858.JavaMail.root@zimbra>

Marcin, I took the liberty of poking around your work dir on fusion.

The problem seems to be that qsub is rejecting the job that swift s submitting to it:

Caused by: Cannot submit job: Could not submit job (qsub reported an exit code of 1). no error output

Now we need to find out why that is.

I see that your tc.data file does not end in a newline. Lets try to get rid of the message about the ":" to eliminate that as a possibility.

Can you also do these things:

- do a quick qsub test of a "echo hi" script to ensure that your Fusion PBS project is still valid, and the qsub is working for you. In this test, set a max wall time the same as what youre trying to set via tc.data (but which I think is being ignored because Swift is unable to parse the GLOBUS namespace declaration from that line)

- see if there are recent files under $HOME/.globus/scripts or other directories under .globus (which I cannot access) which may contain a clue as to why PBS rejecting the job.

- Mike

----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:

> Hi Mike,
> 
> When I look at tc.data it seems to be fine (I made sure and ran again
> and
> got the same error). I also have not changed tc.data since I ran it
> last
> and I seem to remember getting the same error about the illegal
> character
> before.
> 
> I am quite sure I haven't changed anything since I ran last, so I am
> wondering if it might be some changes in fusion which I need to
> update
> for?
> 
...

> >
> > I see two problems, the second likely being the result of the
> first:
> >
> > In your output file: [ERROR] Parsing profiles on line 21 Illegal
> character
> > ':'at position 22 :Illegal character ':'
> > is referring to your tc.data file. I think your Globus MaxWallTime
> profile
> > entry got moved to a separate line, instead of being separated by
> tabs as
> > the last column of the previous line.
> >
> > I suspect that may have caused jobs to get submitted to PBS with
> defaults
> > that were invalid for the default queue that your jobs are going
> into,
> > thus causing the second error: Cannot submit job: Could not submit
> job
> > (qsub reported an exit code of 1). no error output
> >
> > So fix tc.data, and see if this fixes the problem.
> >
> > - Mike
> >
> > ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >
> >> Hi Mike,
> >>
> >> Thanks for your response. I am attaching the .log file. Also, the
> >> swift.out file I included in the original email has the output of
> my
> >> run.
> >> I am including it again.
> >>
> >> Best,
> >>
> >> Marcin
> >>
> >> > Marcin, I forgot to point out: "failed to transfer wrapper log"
> is
> >> just a
> >> > catch-all error message which means "something went wrong with
> an
> >> app()
> >> > job that Swift ran, and the job did not return the expected log
> file
> >> that
> >> > comes from the wrapper script under which Swift runs the job. 
> We
> >> need to
> >> > improve the text of this message.
> >> >
> >> > Also, if you can, always run swift with standard output and
> error
> >> > redirected into a file, and send that file as well when you
> report
> >> a
> >> > problem.
> >> >
> >> > Thanks,
> >> >
> >> > Mike
> >> >
> >> > ----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> >> >
> >> >> Marcin, can you also post the .log file from this run? (it will
> be
> >> >> named getcoefs*.log where * is a long unique id including the
> >> date)
> >> >>
> >> >> - Mike
> >> >>
> >> >> ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I am trying to run swift scripts on fusion and I am
> encountering
> >> an
> >> >> > error
> >> >> > I have never had before (The scripts I am running have worked
> >> >> before).
> >> >> > It
> >> >> > seems it is having trouble submitting job because it "failed
> to
> >> >> > transfer
> >> >> > wrapper log". I am including my swift script, tc.data,
> sites.xml
> >> >> and
> >> >> > the
> >> >> > output file of when I ran it.
> >> >> >
> >> >> > I am not sure if I need to change anything? Like I said, I am
> >> sure
> >> >> > the
> >> >> > same script worked a month or so ago.
> >> >> >
> >> >> > Thanks for your help.
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > Marcin
> >> >> >
> >> >> > _______________________________________________
> >> >> > Swift-user mailing list
> >> >> > Swift-user at ci.uchicago.edu
> >> >> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >> >>
> >> >> --
> >> >> Michael Wilde
> >> >> Computation Institute, University of Chicago
> >> >> Mathematics and Computer Science Division
> >> >> Argonne National Laboratory
> >> >>
> >> >> _______________________________________________
> >> >> Swift-user mailing list
> >> >> Swift-user at ci.uchicago.edu
> >> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >> >
> >> > --
> >> > Michael Wilde
> >> > Computation Institute, University of Chicago
> >> > Mathematics and Computer Science Division
> >> > Argonne National Laboratory
> >> >
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 13 12:52:07 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 13 Mar 2010 12:52:07 -0600 (CST)
Subject: [Swift-user] cannot submit on fusion
In-Reply-To: <15545244.2871268505903393.JavaMail.root@zimbra>
Message-ID: <18579904.2921268506327085.JavaMail.root@zimbra>


----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:

> Hi Mike,
> 
> So, I think I know what it might be. It seems that the two projects I
> am
> listed under do not have any cpu core time available, which would
> explain
> why qsub is not working. So, I guess the problem has nothing to do
> with
> swift.

Cool. Can you get more hours added?  Want to try running some stuff at CI? (perhaps we can do this transparently), by extending the sites.xml below.

> I actually have one other question. When I do finally submit jobs,
> will
> swift make use of the fact that I have 8 processors on each node? Is
> there
> anything I need to add to sites.xml or tc.data so that I do not waste
> any
> available processors, as I only need one per job?

Good point. This depends a lot on how PBS is treating your job specs, which I must admit I still have some unanswered questions regarding.

*If* PBS is treating a request for 1 node as a request for 1 core, it will assign multiple 1-core job requests to the same 8-core node. qstat -n and qstat -f will help us determine whats happening; also reading the Fusion online info and asking a question on its support list.

We can guarantee whats happening by using Swift "coasters" using the sites.xml entry below, and thus more precisely craft how your jobs will be mapped to PBS:

  <pool handle="pbs">
    <execution provider="coaster" url="none" jobmanager="local:pbs"/>

    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="maxTime">3500</profile>         <!-- ADJUST -->
    <profile namespace="globus" key="maxWallTime">00:00:30</profile> <!-- ADJUST -->

    <!-- Run up to 8 PBS jobs of 4 nodes each (up to 256 cores total): -->
    <profile namespace="globus" key="slots">8</profile>
    <profile namespace="globus" key="nodeGranularity">4</profile>
    <profile namespace="globus" key="maxNodes">4</profile>

    <!-- run up to 256 app() tasks at once: (2.55*100)+1 -->
    <profile namespace="karajan" key="jobThrottle">2.55</profile>   
    <profile namespace="karajan" key="initialScore">10000</profile>

    <filesystem provider="local" url="none"/>
    <workdirectory>/home/wilde/swiftwork</workdirectory>            <!-- ADJUST -->
  </pool>

Do you want to try to replicate your data to PADS to try something similar there?

Also, do you perchance have any Grid certificates (TeraGrid, OSG/DOEGrids)? If so, you can leave your data in place and run on PADS and/or TeraPort. If not, I'll see if we can configure coasters to use a dummy certificate to enable you to run coasters over ssh on remote machines.

- Mike

> Thanks again for the help. Please let me know if you would like more
> detail about my swift use. I am happy to help and give feedback.
> 
> Marcin
> 
> > Marcin, I took the liberty of poking around your work dir on
> fusion.
> >
> > The problem seems to be that qsub is rejecting the job that swift s
> > submitting to it:
> >
> > Caused by: Cannot submit job: Could not submit job (qsub reported an
> exit
> > code of 1). no error output
> >
> > Now we need to find out why that is.
> >
> > I see that your tc.data file does not end in a newline. Lets try to
> get
> > rid of the message about the ":" to eliminate that as a
> possibility.
> >
> > Can you also do these things:
> >
> > - do a quick qsub test of a "echo hi" script to ensure that your
> Fusion
> > PBS project is still valid, and the qsub is working for you. In this
> test,
> > set a max wall time the same as what youre trying to set via tc.data
> (but
> > which I think is being ignored because Swift is unable to parse the
> GLOBUS
> > namespace declaration from that line)
> >
> > - see if there are recent files under $HOME/.globus/scripts or
> other
> > directories under .globus (which I cannot access) which may contain
> a clue
> > as to why PBS rejecting the job.
> >
> > - Mike
> >
> > ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >
> >> Hi Mike,
> >>
> >> When I look at tc.data it seems to be fine (I made sure and ran
> again
> >> and
> >> got the same error). I also have not changed tc.data since I ran
> it
> >> last
> >> and I seem to remember getting the same error about the illegal
> >> character
> >> before.
> >>
> >> I am quite sure I haven't changed anything since I ran last, so I
> am
> >> wondering if it might be some changes in fusion which I need to
> >> update
> >> for?
> >>
> > ...
> >
> >> >
> >> > I see two problems, the second likely being the result of the
> >> first:
> >> >
> >> > In your output file: [ERROR] Parsing profiles on line 21 Illegal
> >> character
> >> > ':'at position 22 :Illegal character ':'
> >> > is referring to your tc.data file. I think your Globus
> MaxWallTime
> >> profile
> >> > entry got moved to a separate line, instead of being separated
> by
> >> tabs as
> >> > the last column of the previous line.
> >> >
> >> > I suspect that may have caused jobs to get submitted to PBS with
> >> defaults
> >> > that were invalid for the default queue that your jobs are going
> >> into,
> >> > thus causing the second error: Cannot submit job: Could not
> submit
> >> job
> >> > (qsub reported an exit code of 1). no error output
> >> >
> >> > So fix tc.data, and see if this fixes the problem.
> >> >
> >> > - Mike
> >> >
> >> > ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >> >
> >> >> Hi Mike,
> >> >>
> >> >> Thanks for your response. I am attaching the .log file. Also,
> the
> >> >> swift.out file I included in the original email has the output
> of
> >> my
> >> >> run.
> >> >> I am including it again.
> >> >>
> >> >> Best,
> >> >>
> >> >> Marcin
> >> >>
> >> >> > Marcin, I forgot to point out: "failed to transfer wrapper
> log"
> >> is
> >> >> just a
> >> >> > catch-all error message which means "something went wrong
> with
> >> an
> >> >> app()
> >> >> > job that Swift ran, and the job did not return the expected
> log
> >> file
> >> >> that
> >> >> > comes from the wrapper script under which Swift runs the job.
> >> We
> >> >> need to
> >> >> > improve the text of this message.
> >> >> >
> >> >> > Also, if you can, always run swift with standard output and
> >> error
> >> >> > redirected into a file, and send that file as well when you
> >> report
> >> >> a
> >> >> > problem.
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Mike
> >> >> >
> >> >> > ----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:
> >> >> >
> >> >> >> Marcin, can you also post the .log file from this run? (it
> will
> >> be
> >> >> >> named getcoefs*.log where * is a long unique id including
> the
> >> >> date)
> >> >> >>
> >> >> >> - Mike
> >> >> >>
> >> >> >> ----- "Marcin Hitczenko" <marcin at galton.uchicago.edu> wrote:
> >> >> >>
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > I am trying to run swift scripts on fusion and I am
> >> encountering
> >> >> an
> >> >> >> > error
> >> >> >> > I have never had before (The scripts I am running have
> worked
> >> >> >> before).
> >> >> >> > It
> >> >> >> > seems it is having trouble submitting job because it
> "failed
> >> to
> >> >> >> > transfer
> >> >> >> > wrapper log". I am including my swift script, tc.data,
> >> sites.xml
> >> >> >> and
> >> >> >> > the
> >> >> >> > output file of when I ran it.
> >> >> >> >
> >> >> >> > I am not sure if I need to change anything? Like I said, I
> am
> >> >> sure
> >> >> >> > the
> >> >> >> > same script worked a month or so ago.
> >> >> >> >
> >> >> >> > Thanks for your help.
> >> >> >> >
> >> >> >> > Best,
> >> >> >> >
> >> >> >> > Marcin
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Swift-user mailing list
> >> >> >> > Swift-user at ci.uchicago.edu
> >> >> >> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >> >> >>
> >> >> >> --
> >> >> >> Michael Wilde
> >> >> >> Computation Institute, University of Chicago
> >> >> >> Mathematics and Computer Science Division
> >> >> >> Argonne National Laboratory
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> Swift-user mailing list
> >> >> >> Swift-user at ci.uchicago.edu
> >> >> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >> >> >
> >> >> > --
> >> >> > Michael Wilde
> >> >> > Computation Institute, University of Chicago
> >> >> > Mathematics and Computer Science Division
> >> >> > Argonne National Laboratory
> >> >> >
> >> >
> >> > --
> >> > Michael Wilde
> >> > Computation Institute, University of Chicago
> >> > Mathematics and Computer Science Division
> >> > Argonne National Laboratory
> >> >
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.uchicago.edu  Mon Mar 15 12:49:18 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Mon, 15 Mar 2010 12:49:18 -0500
Subject: [Swift-user] CFP deadline 3/17/10: IEEE 2010 Fourth International
 Workshop on Scientific Workflows (SWF 2010)
Message-ID: <4B9E731E.4070206@cs.uchicago.edu>

CALL FOR PAPERS
IEEE 2010 Fourth International Workshop on Scientific Workflows (SWF 2010)
http://www.cs.wayne.edu/~shiyong/swf
Miama, Florida, U.S.A., one day between July 5-10, 2010

In conjunction with IEEE ICWS 2010, IEEE SCC 2010, IEEE CLOUD 2010, and 
IEEE SERVICES 2010

Description
Scientific workflows have become an increasingly popular paradigm for 
scientists to formalize and structure complex scientific processes to 
enable and accelerate many significant scientific discoveries. A 
scientific workflow is a formal specification of a scientific process, 
which represents, streamlines, and automates the analytical and 
computational steps that a scientist needs to go through from dataset 
selection and integration, computation and analysis, to final data 
product presentation and visualization. A scientific workflow management 
system (SWFMS) is a system that supports the specification, 
modification, execution, failure handling, and monitoring of a 
scientific workflow using the workflow logic to control the order of 
executing workflow tasks. The importance of scientific workflows has 
been recognized by NSF since 2006 and was reemphasized recently in an 
science article titled ?Beyond the Data Deluge? (Science, Vol. 323. no. 
5919, pp. 1297 ? 1298, 2009), which concluded, ?In the future, the 
rapidity with which any given discipline advances is likely to depend on 
how well the community acquires the necessary expertise in database, 
workflow management, visualization, and cloud computing technologies.?

The goal of SWF 2010 is to provide a forum for researchers and 
practitioners to present their recent research results and best 
practices of scientific workflows, and identify the emerging trends, 
opportunities, problems, and challenges in this area. Authors are 
invited to submit regular papers (8 pages) and short papers (4 pages) 
that show original unpublished research results in all areas of 
scientific workflows. Topics of interest are listed below; however, 
submissions on all aspects of scientific workflows are welcome.

List of topics
* Scientific workflow provenance management and analytics
* Scientific workflow data, metadata, service, and task management
* Scientific workflow architectures, models, languages, systems, and 
algorithms
* Scientific workflow monitoring, debugging, and failure handling
* Streaming data processing in scientific workflows
* Pipelined, data, workflow, and task parallelism in scientific workflows
* Cloud, Service, Grid, or hybrid scientific workflows
* Data, metadata, compute, user-interaction, or visualization-intensive 
scientific workflows
* Scientific workflow composition
* Security issues in scientific workflows
* Data integration and service integration in scientific workflows
* Scientific workflow mapping, optimization, and scheduling
* Scientific workflow modeling, simulation, analysis, and verification
* Scalability, reliability, extensibility, agility, and interoperability
* Scientific workflow applications

Important dates
* Paper Submission March 17, 2010
* Decision Notification (Electronic) April 17, 2010
* Camera-Ready Submission & Pre-registration April 30, 2010

Paper submission
All papers should be submitted via the SWF workshop submission system at 
http://www.confhub.com/conf.php?id=175. First time users need to 
register with the system first.

Workshop chairs
* Shiyong Lu, Wayne State University, Email: shiyong at wayne.edu
* Calton Pu, Georgia Tech
* Liqiang Wang, University of Wyoming

Publication chairs
* Ilkay Altintas, San Diego Supercomputer Center
* Yogesh Simmhan, Microsoft Research
* Ioan Raicu, Northwestern University

Publicity chair
* Jamal Alhiyafi, Wayne State University, Email: alhiyafi at wayne.edu

Program committee
* Ilkay Altintas, San Diego Supercomputer Center, USA
* Roger Barga, Microsoft Research, USA
* Adam Barker, University of Oxford, UK
* Shawn Bowers, UC Davis Genome Center, USA
* Artem Chebotko, University of Texas at Pan American, USA
* Ian Gorton, PNNL
* Paul Groth, VU University Amsterdam
* Marta L. Queir?s Mattoso, Federal University of Rio de Janeiro, Brazil
* Luc Moreau, University of South Hampton
* Ioan Raicu, University of Chicago, USA
* Yogesh Simmhan, Microsoft Corporation, USA
* Chung-Wei Hang, North Carolina State University, USA
* Hasan Jamil, Wayne State University
* Ian Taylor, Cardiff University, UK
* Jianwu Wang, San Diego Supercomputer Center
* Wei Tan, ANL
* Ping Yang, Binghamton University, USA
* Ustun Yildiz, UC Davis
* Yong Zhao, Microsoft Corporation, USA
* Zhiming Zhao, University of Amsterdam, the Netherlands

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================



-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




From yizhu at cs.uchicago.edu  Thu Mar 18 12:21:19 2010
From: yizhu at cs.uchicago.edu (Yi Zhu)
Date: Thu, 18 Mar 2010 12:21:19 -0500
Subject: [Swift-user] get error when try to set swiftworkdirectory to /gpfs
Message-ID: <4BA2610F.9030502@cs.uchicago.edu>

Hi,

I have a problem after switch my swift working directory from /home/ to 
/gpfs

My jobs was running on PADS, and my input data was stored at /gpfs and 
the swift work directory was set to /home/swiftwork and everything works 
fine.  But when I change my swift working directory from 
/home/yizhu/swiftwork ( in sites.xml) to /gpfs/ci/swift/yizhu, i got 
lots of job failure( about half of them failed).

since /gpfs/ is also a shared file system and should be transparent for 
swift, I have no idea why this happened.

-Yi Zhu



From yizhu at cs.uchicago.edu  Thu Mar 18 16:47:13 2010
From: yizhu at cs.uchicago.edu (Yi Zhu)
Date: Thu, 18 Mar 2010 16:47:13 -0500
Subject: [Swift-user] Fwd: get error when try to set swiftworkdirectory to
	/gpfs
Message-ID: <4BA29F61.4090903@cs.uchicago.edu>


Hi

I think I've found the reason of error, by using direct address rather 
than linked address, the error disappeared.

e.g. instead of set the directory address: 
"/gpfs/ci/swift/yizhu/swiftwork", i use 
"/autonfs/gpfs-pads/projects/CI-CCR000013/yizhu/swiftwork", and then 
everything works fine.

-Yi
-------- Original Message --------
Subject: 	get error when try to set swiftworkdirectory to /gpfs
Date: 	Thu, 18 Mar 2010 12:21:19 -0500
From: 	Yi Zhu <yizhu at cs.uchicago.edu>
To: 	swift-user at ci.uchicago.edu



Hi,

I have a problem after switch my swift working directory from /home/ to
/gpfs

My jobs was running on PADS, and my input data was stored at /gpfs and
the swift work directory was set to /home/swiftwork and everything works
fine.  But when I change my swift working directory from
/home/yizhu/swiftwork ( in sites.xml) to /gpfs/ci/swift/yizhu, i got
lots of job failure( about half of them failed).

since /gpfs/ is also a shared file system and should be transparent for
swift, I have no idea why this happened.

-Yi Zhu


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100318/de7dd076/attachment.html>

From wilde at mcs.anl.gov  Fri Mar 19 09:37:04 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 19 Mar 2010 09:37:04 -0500 (CDT)
Subject: [Swift-user] Fwd: get error when try to set swiftworkdirectory
	to /gpfs
In-Reply-To: <4BA29F61.4090903@cs.uchicago.edu>
Message-ID: <12374462.159951269009424440.JavaMail.root@zimbra>

Yi, Ti checked all the nodes, and the symlinks to /gpfs were all set correctly.

Can you point us to the log file(s) for the runs that failed in this manner?

Thanks,

Mike


----- "Yi Zhu" <yizhu at cs.uchicago.edu> wrote:

> Hi
> 
> I think I've found the reason of error, by using direct address rather
> than linked address, the error disappeared.
> 
> e.g. instead of set the directory address:
> "/gpfs/ci/swift/yizhu/swiftwork", i use
> "/autonfs/gpfs-pads/projects/CI-CCR000013/yizhu/swiftwork", and then
> everything works fine.
> 
> -Yi
> -------- Original Message --------
> Subject: 	get error when try to set swiftworkdirectory to /gpfs
> Date: 	Thu, 18 Mar 2010 12:21:19 -0500
> From: 	Yi Zhu <yizhu at cs.uchicago.edu>
> To: 	swift-user at ci.uchicago.edu
> 
> Hi,
> 
> I have a problem after switch my swift working directory from /home/
> to
> /gpfs
> 
> My jobs was running on PADS, and my input data was stored at /gpfs and
> the swift work directory was set to /home/swiftwork and everything
> works
> fine.  But when I change my swift working directory from
> /home/yizhu/swiftwork ( in sites.xml) to /gpfs/ci/swift/yizhu, i got
> lots of job failure( about half of them failed).
> 
> since /gpfs/ is also a shared file system and should be transparent
> for
> swift, I have no idea why this happened.
> 
> -Yi Zhu 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.uchicago.edu  Fri Mar 19 14:08:36 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Mar 2010 14:08:36 -0500
Subject: [Swift-user] CFP deadline extension to 3/24/10: IEEE 2010 Fourth
 International Workshop on Scientific Workflows (SWF 2010)
Message-ID: <4BA3CBB4.2080807@cs.uchicago.edu>

CALL FOR PAPERS
IEEE 2010 Fourth International Workshop on Scientific Workflows (SWF 2010)
http://www.cs.wayne.edu/~shiyong/swf
Miama, Florida, U.S.A., one day between July 5-10, 2010

In conjunction with IEEE ICWS 2010, IEEE SCC 2010, IEEE CLOUD 2010, and 
IEEE SERVICES 2010

Description
Scientific workflows have become an increasingly popular paradigm for 
scientists to formalize and structure complex scientific processes to 
enable and accelerate many significant scientific discoveries. A 
scientific workflow is a formal specification of a scientific process, 
which represents, streamlines, and automates the analytical and 
computational steps that a scientist needs to go through from dataset 
selection and integration, computation and analysis, to final data 
product presentation and visualization. A scientific workflow management 
system (SWFMS) is a system that supports the specification, 
modification, execution, failure handling, and monitoring of a 
scientific workflow using the workflow logic to control the order of 
executing workflow tasks. The importance of scientific workflows has 
been recognized by NSF since 2006 and was reemphasized recently in an 
science article titled "Beyond the Data Deluge" (Science, Vol. 323. no. 
5919, pp. 1297 -- 1298, 2009), which concluded, "In the future, the 
rapidity with which any given discipline advances is likely to depend on 
how well the community acquires the necessary expertise in database, 
workflow management, visualization, and cloud computing technologies."

The goal of SWF 2010 is to provide a forum for researchers and 
practitioners to present their recent research results and best 
practices of scientific workflows, and identify the emerging trends, 
opportunities, problems, and challenges in this area. Authors are 
invited to submit regular papers (8 pages) and short papers (4 pages) 
that show original unpublished research results in all areas of 
scientific workflows. Topics of interest are listed below; however, 
submissions on all aspects of scientific workflows are welcome.

List of topics
* Scientific workflow provenance management and analytics
* Scientific workflow data, metadata, service, and task management
* Scientific workflow architectures, models, languages, systems, and 
algorithms
* Scientific workflow monitoring, debugging, and failure handling
* Streaming data processing in scientific workflows
* Pipelined, data, workflow, and task parallelism in scientific workflows
* Cloud, Service, Grid, or hybrid scientific workflows
* Data, metadata, compute, user-interaction, or visualization-intensive 
scientific workflows
* Scientific workflow composition
* Security issues in scientific workflows
* Data integration and service integration in scientific workflows
* Scientific workflow mapping, optimization, and scheduling
* Scientific workflow modeling, simulation, analysis, and verification
* Scalability, reliability, extensibility, agility, and interoperability
* Scientific workflow applications

Important dates
* Paper Submission Extended to March 24, 2010
* Decision Notification (Electronic) April 17, 2010
* Camera-Ready Submission & Pre-registration April 30, 2010

Paper submission
All papers should be submitted via the SWF workshop submission system at 
http://www.confhub.com/conf.php?id=175. First time users need to 
register with the system first.

Workshop chairs
* Shiyong Lu, Wayne State University, Email: shiyong at wayne.edu
* Calton Pu, Georgia Tech
* Liqiang Wang, University of Wyoming

Publication chairs
* Ilkay Altintas, San Diego Supercomputer Center
* Yogesh Simmhan, Microsoft Research
* Ioan Raicu, Northwestern University

Publicity chair
* Jamal Alhiyafi, Wayne State University, Email: alhiyafi at wayne.edu

Program committee
* Ilkay Altintas, San Diego Supercomputer Center, USA
* Roger Barga, Microsoft Research, USA
* Adam Barker, University of Oxford, UK
* Shawn Bowers, UC Davis Genome Center, USA
* Artem Chebotko, University of Texas at Pan American, USA
* Ian Gorton, PNNL
* Paul Groth, VU University Amsterdam
* Marta L. Queir?s Mattoso, Federal University of Rio de Janeiro, Brazil
* Luc Moreau, University of South Hampton
* Ioan Raicu, University of Chicago, USA
* Yogesh Simmhan, Microsoft Corporation, USA
* Chung-Wei Hang, North Carolina State University, USA
* Hasan Jamil, Wayne State University
* Ian Taylor, Cardiff University, UK
* Jianwu Wang, San Diego Supercomputer Center
* Wei Tan, ANL
* Ping Yang, Binghamton University, USA
* Ustun Yildiz, UC Davis
* Yong Zhao, Microsoft Corporation, USA
* Zhiming Zhao, University of Amsterdam, the Netherlands

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
      https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100319/25c4ae90/attachment.html>

From vipulkrsingh at gmail.com  Sat Mar 20 14:42:26 2010
From: vipulkrsingh at gmail.com (Vipul Kumar Singh)
Date: Sun, 21 Mar 2010 01:12:26 +0530
Subject: [Swift-user] running swift .... on ssh
Message-ID: <65fb392f1003201242i290a95e0k1ffa483ba3d9638e@mail.gmail.com>

hi

I have two system with swift setup (as per instruction on downloads page of
http://www.ci.uchicago.edu/swift );
Both systems have user accounts with same name (globus) and configured to
login via ssh without password..

system1 is 192.168.56.102 globus1.vitu.org
system2 is 192.168.56.103 globus2.vitu.org

I get following error doing : swift first.swift

Swift svn swift-r3264 cog-r2730

RunID: 20100321-0050-e4cat3jf
Progress:
Execution failed:
    Could not find any valid host for task "Task(type=UNKNOWN,
identity=urn:cog-
1269112861163)" with constraints {tr=echo,
filenames=[Ljava.lang.String;@5e1077, trfqn=echo,
filecache=org.griphyn.vdl.karajan.lib.cache.CacheMapAdapter at 181ed9e}


sites.xml  :-------------------------------------

<config>
  <pool handle="globus1">
    <filesystem url="globus1.vitu.org" provider="ssh" />
    <execution url="globus1.vitu.org" provider="ssh" />
    <workdirectory >/home/globus/tmp</workdirectory>
    <profile namespace="karajan" key="jobThrottle">0</profile>
  </pool>


  <pool handle="globus2">
    <filesystem url="globus2.vitu.org" provider="ssh" />
    <execution url="globus2.vitu.org" provider="ssh" />
    <workdirectory >/home/globus/tmp</workdirectory>
    <profile namespace="karajan" key="jobThrottle">0</profile>
  </pool>
</config>



Haven't changed any thing in swift.properties except

hostname=globus1.vitu.org

tc.data -----------------------------------------------
#This is the transformation catalog.
#
#It comes pre-configured with a number of simple transformations with
#paths that are likely to work on a linux box. However, on some systems,
#the paths to these executables will be different (for example, sometimes
#some of these programs are found in /usr/bin rather than in /bin)
#
#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
#there must be no trailing whitespace at the end of each line.
#
# sitename  transformation  path   INSTALLED  platform  profiles
localhost     echo         /bin/echo    INSTALLED    INTEL32::LINUX    null
localhost     cat         /bin/cat    INSTALLED    INTEL32::LINUX    null
localhost     ls         /bin/ls        INSTALLED    INTEL32::LINUX    null
localhost     grep         /bin/grep    INSTALLED    INTEL32::LINUX    null
localhost     sort         /bin/sort    INSTALLED    INTEL32::LINUX    null
localhost     paste         /bin/paste    INSTALLED    INTEL32::LINUX
null


swift.log
file--------------------------------------------------------------------------------
2010-03-19 21:48:28,359+0530 DEBUG Loader Loader started
2010-03-19 21:48:40,308+0530 DEBUG Loader Loader started
2010-03-19 21:52:45,812+0530 DEBUG Loader Loader started
2010-03-20 23:28:16,738+0530 DEBUG Loader Loader started
2010-03-20 23:33:52,976+0530 DEBUG Loader Loader started
2010-03-20 23:37:05,687+0530 DEBUG Loader Loader started
2010-03-20 23:38:45,994+0530 DEBUG Loader Loader started
2010-03-21 00:23:50,310+0530 DEBUG Loader Loader started
2010-03-21 00:25:05,131+0530 DEBUG Loader Loader started
2010-03-21 00:49:47,095+0530 DEBUG Loader Loader started
2010-03-21 00:50:46,013+0530 DEBUG Loader Loader started

Vipul Kumar Singh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100321/3922a6a2/attachment.html>

From vipulkrsingh at gmail.com  Sat Mar 20 16:50:47 2010
From: vipulkrsingh at gmail.com (Vipul Kumar Singh)
Date: Sun, 21 Mar 2010 03:20:47 +0530
Subject: [Swift-user] swift on multiple pcs
Message-ID: <65fb392f1003201450n72165402u690748bda7b67780@mail.gmail.com>

hi,

(i) I have 4 ubuntu installations in virtual box..

(ii) all have swift and can log in with ssh on each other...

(iii) To make swift submit job to other node, is it required to setup any
thing else (other than configuring sites.xml and swift.properties) ..

i have just followed the steps to build stable swift from downloads section
of swift site...

Thank you

Vipul Kumar Singh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100321/4ec8f126/attachment.html>

From wilde at mcs.anl.gov  Sat Mar 20 18:26:24 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 20 Mar 2010 18:26:24 -0500 (CDT)
Subject: [Swift-user] running swift .... on ssh
In-Reply-To: <11483913.191861269120833836.JavaMail.root@zimbra>
Message-ID: <15968114.192781269127584423.JavaMail.root@zimbra>

Vipul,

You need to add an entry to tc.data for the app "echo" on your two "sites", globus1 and globus2. The message is saying Swift is trying to run app "echo" and there are no sites listed in tc.data where that app exists.

Easiest way: copy swifts's etc/tc.data file to the directory in which you are running the swift command, and replicate the localhost entry for echo twice, for these two test sites.

Be sure to use *tabs* as the separator; the parser is picky for historical reasons (to be fixed).  

Then use the "swift -tc.file tc.data" option to make swift use your updated tc.

- Mike


----- "Vipul Kumar Singh" <vipulkrsingh at gmail.com> wrote:

> hi
> 
> I have two system with swift setup (as per instruction on downloads
> page of http://www.ci.uchicago.edu/swift );
> Both systems have user accounts with same name (globus) and configured
> to login via ssh without password..
> 
> system1 is 192.168.56.102 globus1.vitu.org
> system2 is 192.168.56.103 globus2.vitu.org
> 
> I get following error doing : swift first.swift
> 
> Swift svn swift-r3264 cog-r2730
> 
> RunID: 20100321-0050-e4cat3jf
> Progress:
> Execution failed:
> Could not find any valid host for task "Task(type=UNKNOWN,
> identity=urn:cog-
> 1269112861163)" with constraints {tr=echo,
> filenames=[Ljava.lang.String;@5e1077, trfqn=echo,
> filecache=org.griphyn.vdl.karajan.lib.cache.CacheMapAdapter at 181ed9e}
> 
> 
> sites.xml :-------------------------------------
> 
> <config>
> <pool handle="globus1">
> <filesystem url=" globus1.vitu.org " provider="ssh" />
> <execution url=" globus1.vitu.org " provider="ssh" />
> <workdirectory >/home/globus/tmp</workdirectory>
> <profile namespace="karajan" key="jobThrottle">0</profile>
> </pool>
> 
> 
> <pool handle="globus2">
> <filesystem url=" globus2.vitu.org " provider="ssh" />
> <execution url=" globus2.vitu.org " provider="ssh" />
> <workdirectory >/home/globus/tmp</workdirectory>
> <profile namespace="karajan" key="jobThrottle">0</profile>
> </pool>
> </config>
> 
> 
> 
> Haven't changed any thing in swift.properties except
> 
> hostname= globus1.vitu.org
> 
> tc.data -----------------------------------------------
> #This is the transformation catalog.
> #
> #It comes pre-configured with a number of simple transformations with
> #paths that are likely to work on a linux box. However, on some
> systems,
> #the paths to these executables will be different (for example,
> sometimes
> #some of these programs are found in /usr/bin rather than in /bin)
> #
> #NOTE WELL: fields in this file must be separated by tabs, not spaces;
> and
> #there must be no trailing whitespace at the end of each line.
> #
> # sitename transformation path INSTALLED platform profiles
> localhost echo /bin/echo INSTALLED INTEL32::LINUX null
> localhost cat /bin/cat INSTALLED INTEL32::LINUX null
> localhost ls /bin/ls INSTALLED INTEL32::LINUX null
> localhost grep /bin/grep INSTALLED INTEL32::LINUX null
> localhost sort /bin/sort INSTALLED INTEL32::LINUX null
> localhost paste /bin/paste INSTALLED INTEL32::LINUX null
> 
> 
> swift.log
> file--------------------------------------------------------------------------------
> 2010-03-19 21:48:28,359+0530 DEBUG Loader Loader started
> 2010-03-19 21:48:40,308+0530 DEBUG Loader Loader started
> 2010-03-19 21:52:45,812+0530 DEBUG Loader Loader started
> 2010-03-20 23:28:16,738+0530 DEBUG Loader Loader started
> 2010-03-20 23:33:52,976+0530 DEBUG Loader Loader started
> 2010-03-20 23:37:05,687+0530 DEBUG Loader Loader started
> 2010-03-20 23:38:45,994+0530 DEBUG Loader Loader started
> 2010-03-21 00:23:50,310+0530 DEBUG Loader Loader started
> 2010-03-21 00:25:05,131+0530 DEBUG Loader Loader started
> 2010-03-21 00:49:47,095+0530 DEBUG Loader Loader started
> 2010-03-21 00:50:46,013+0530 DEBUG Loader Loader started
> 
> Vipul Kumar Singh
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 20 18:26:26 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 20 Mar 2010 18:26:26 -0500 (CDT)
Subject: [Swift-user] swift on multiple pcs
In-Reply-To: <65fb392f1003201450n72165402u690748bda7b67780@mail.gmail.com>
Message-ID: <30538418.192811269127586601.JavaMail.root@zimbra>


----- "Vipul Kumar Singh" <vipulkrsingh at gmail.com> wrote:

> hi,
> 
> (i) I have 4 ubuntu installations in virtual box..
> 
> (ii) all have swift and can log in with ssh on each other...
> 
> (iii) To make swift submit job to other node, is it required to setup
> any thing else (other than configuring sites.xml and swift.properties)

and make sure you put valid entries in tc.data for all app() application names you want to run on those nodes.

Make each node a "site" (ie a <pool> element) in sites.xml.

Make sure the <workdirectory> element in tc.data points to a valid directory on those machines.

And to use the ssh provider, set $HOME/.ssh/app.defaults appropriately.

Ive started to place some examples of these files 
> ..
> 
> i have just followed the steps to build stable swift from downloads
> section of swift site...
> 
> Thank you
> 
> Vipul Kumar Singh
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 20 18:31:15 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 20 Mar 2010 18:31:15 -0500 (CDT)
Subject: [Swift-user] swift on multiple pcs
In-Reply-To: <23899838.192841269127710036.JavaMail.root@zimbra>
Message-ID: <2351660.192891269127875574.JavaMail.root@zimbra>

Sorry, I hit send before I finished this:

----- "Vipul Kumar Singh" <vipulkrsingh at gmail.com> wrote:

> hi,
> 
> (i) I have 4 ubuntu installations in virtual box..
> 
> (ii) all have swift and can log in with ssh on each other...
> 
> (iii) To make swift submit job to other node, is it required to setup
> any thing else (other than configuring sites.xml and swift.properties)

and make sure you put valid entries in tc.data for all app() application names you want to run on those nodes.

Make each node a "site" (ie a <pool> element) in sites.xml.

Make sure the <workdirectory> element in tc.data points to a valid directory on those machines.

And to use the ssh provider, set $HOME/.ssh/app.defaults appropriately.

Ive started to place some examples of these files in a swift "cookbook" where I'm accumulating (very) preliminary text that will go into the Users Guide:

 http://www.ci.uchicago.edu/wiki/bin/view/SWFT/SwiftCookBook#SItes_xml_ssh_provider_settings

- Mike

> ..
> 
> i have just followed the steps to build stable swift from downloads
> section of swift site...
> 
> Thank you
> 
> Vipul Kumar Singh
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From vipulkrsingh at gmail.com  Sat Mar 20 14:38:43 2010
From: vipulkrsingh at gmail.com (Vipul Kumar Singh)
Date: Sun, 21 Mar 2010 01:08:43 +0530
Subject: [Swift-user] running swift .... on ssh
Message-ID: <65fb392f1003201238t58ce8c73r6bd8f7ee7e66d866@mail.gmail.com>

hi

I have two system with swift setup (as per instruction on downloads page of
http://www.ci.uchicago.edu/swift );
Both systems have user accounts with same name (globus) and configured to
login via ssh without password..

system1 is 192.168.56.102 globus1.vitu.org
system2 is 192.168.56.103 globus2.vitu.org

I get following error doing : swift first.swift

Swift svn swift-r3264 cog-r2730

RunID: 20100321-0050-e4cat3jf
Progress:
Execution failed:
    Could not find any valid host for task "Task(type=UNKNOWN,
identity=urn:cog-1269112861163)" with constraints {tr=echo,
filenames=[Ljava.lang.String;@5e1077, trfqn=echo,
filecache=org.griphyn.vdl.karajan.lib.cache.CacheMapAdapter at 181ed9e}


sites.xml  :-------------------------------------

<config>
  <pool handle="globus1">
    <filesystem url="globus1.vitu.org" provider="ssh" />
    <execution url="globus1.vitu.org" provider="ssh" />
    <workdirectory >/home/globus/tmp</workdirectory>
    <profile namespace="karajan" key="jobThrottle">0</profile>
  </pool>


  <pool handle="globus2">
    <filesystem url="globus2.vitu.org" provider="ssh" />
    <execution url="globus2.vitu.org" provider="ssh" />
    <workdirectory >/home/globus/tmp</workdirectory>
    <profile namespace="karajan" key="jobThrottle">0</profile>
  </pool>
</config>



Haven't changed any thing in swift.properties except

hostname=globus1.vitu.org

tc.data -----------------------------------------------
#This is the transformation catalog.
#
#It comes pre-configured with a number of simple transformations with
#paths that are likely to work on a linux box. However, on some systems,
#the paths to these executables will be different (for example, sometimes
#some of these programs are found in /usr/bin rather than in /bin)
#
#NOTE WELL: fields in this file must be separated by tabs, not spaces; and
#there must be no trailing whitespace at the end of each line.
#
# sitename  transformation  path   INSTALLED  platform  profiles
localhost     echo         /bin/echo    INSTALLED    INTEL32::LINUX    null
localhost     cat         /bin/cat    INSTALLED    INTEL32::LINUX    null
localhost     ls         /bin/ls        INSTALLED    INTEL32::LINUX    null
localhost     grep         /bin/grep    INSTALLED    INTEL32::LINUX    null
localhost     sort         /bin/sort    INSTALLED    INTEL32::LINUX    null
localhost     paste         /bin/paste    INSTALLED    INTEL32::LINUX
null


swift.log
file--------------------------------------------------------------------------------
2010-03-19 21:48:28,359+0530 DEBUG Loader Loader started
2010-03-19 21:48:40,308+0530 DEBUG Loader Loader started
2010-03-19 21:52:45,812+0530 DEBUG Loader Loader started
2010-03-20 23:28:16,738+0530 DEBUG Loader Loader started
2010-03-20 23:33:52,976+0530 DEBUG Loader Loader started
2010-03-20 23:37:05,687+0530 DEBUG Loader Loader started
2010-03-20 23:38:45,994+0530 DEBUG Loader Loader started
2010-03-21 00:23:50,310+0530 DEBUG Loader Loader started
2010-03-21 00:25:05,131+0530 DEBUG Loader Loader started
2010-03-21 00:49:47,095+0530 DEBUG Loader Loader started
2010-03-21 00:50:46,013+0530 DEBUG Loader Loader started

Vipul Kumar Singh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100321/230b93b3/attachment.html>

From wilde at mcs.anl.gov  Sat Mar 20 20:30:23 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 20 Mar 2010 20:30:23 -0500 (CDT)
Subject: [Swift-user] Fwd: Delivery Status Notification (Failure)
In-Reply-To: <fec1351f1002150821r7113a196k69a33388cd1b0cd9@mail.gmail.com>
Message-ID: <9178038.193871269135023138.JavaMail.root@zimbra>

Tibi, this old message just slipped into the list when I cleaned up pending list admin requests just now.

Let us know if its still a pending problem for you. I'm assuming its not.

Thanks,

Mike

----- "Tiberiu Stef-Praun" <tiberiu.stefpraun at gmail.com> wrote:

> Hello Developers/Users
> 
> 
> 
> 
> I am getting the following exception
> 
> Karajan exception: Ambiguous element: property. Possible choices:
> sys:property
> cdm:property
> 
> It probably comes from using the concurrent mapper:
> 
> type statePath{
> file iterState[];
> }
> statePath statePathFiles<concurrent_mapper; prefix=statePathName,
> sufix=".mat">;
> statePathFiles.iterState[0]=initialize();
> 
> 
> I am using swift from SVN
> 
> Any suggestions ?
> Thank you
> Tibi
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Sat Mar 20 20:32:01 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 20 Mar 2010 20:32:01 -0500 (CDT)
Subject: [Swift-user] Re: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu>
Message-ID: <7309358.193901269135121897.JavaMail.root@zimbra>

This old message slipped through by mistake when I cleaned up pending list admin requests.  Please ignore - its been discussed already.

- Mike

----- "Ioan Raicu" <iraicu at eecs.northwestern.edu> wrote:

> But lets bring this back to a more real example. A user wanting to
> express some computations that have some dependencies, would write out
> their computations in some order, expecting their order to be
> preserved because of the dependencies. If you only support single
> assignment on variables (e.g. the data), then an example like the one
> below could never deadlock because the single assignment would be
> violated on the 2nd statement. Perhaps things are more complicated if
> you support multiple assignments per variables, but that is not the
> case for Swift, right?
> 
> I am trying to understand if this deadlock is happening in Swift due
> to some particular implementation detail in Swift (or underlying
> pieces), or is it a fundamental flaw in the DAG based approach with
> single assignment variables? Or is it due to something completely
> different?
> 
> Thanks,
> Ioan
> --
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu Web:
> http://www.eecs.northwestern.edu/~iraicu/
> https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> =================================================================
> 
> Ben Clifford wrote:
> 
> 
> 
> Mathematically, the two different sequences evaluate to different
> values:
> 
> x = y + 1
> y = x + 1
> 
> assuming y = 0, x = 1 I mean in simultaneous equations (linear
> algebra) - in other words, "find
> (through whatever means you care to use) a value of x and y such that
> the
> above two equations are both satisfied" - there is no value of x and y
> that satisfies that. 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wozniak at mcs.anl.gov  Sat Mar 20 21:39:07 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Sat, 20 Mar 2010 21:39:07 -0500 (Central Daylight Time)
Subject: [Swift-user] Fwd: Delivery Status Notification (Failure)
In-Reply-To: <fec1351f1002150821r7113a196k69a33388cd1b0cd9@mail.gmail.com>
References: <fec1351f1002150818s3cabe1d9x90c42a671d5a0e14@mail.gmail.com>
	<0015175cda8c95db2d047fa5f916@google.com>
	<fec1351f1002150821r7113a196k69a33388cd1b0cd9@mail.gmail.com>
Message-ID: <alpine.WNT.2.00.1003202138050.3440@justinwozniak>

Hello Tibi
 	This problem should have been fixed in trunk a while ago.  Please 
try again and let us know if you have any more problems.
 	Thanks
 	Justin

On Mon, 15 Feb 2010, Tiberiu Stef-Praun wrote:

> Hello Developers/Users
>
> I am getting the following exception
>
> Karajan exception: Ambiguous element: property. Possible choices:
>   sys:property
>   cdm:property
>
> It probably comes from using the concurrent mapper:
>
> type statePath{
>       file iterState[];
> }
> statePath statePathFiles<concurrent_mapper; prefix=statePathName,
> sufix=".mat">;
> statePathFiles.iterState[0]=initialize();
>
>
> I am using swift from SVN
>
> Any suggestions ?
> Thank you
> Tibi
>

-- 
Justin M Wozniak


From aespinosa at cs.uchicago.edu  Sun Mar 21 17:21:38 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sun, 21 Mar 2010 17:21:38 -0500
Subject: [Swift-user] last element array indexing?
Message-ID: <50b07b4b1003211521x7d111fd6rad1f7a931b0eef86@mail.gmail.com>

Hi,

How can I refer to the last element of the array?  my external mapper
doesn't know beforehand how many elements it will give the array.

Other languages index this with negative integers.

Sample script:
type File;

File arr[] <ext;exec="arbitrary_mapper.rb">
File out <"foo.txt">;
foo = some_app(arr[-1])  /* doesn't work in swift */

arbitrary_mapper.rb
!/usr/bin/env ruby

(0..rand(20)).each do |i|
    puts "[#{i}] rand#{i}.txt"
end


I can  always workaround this by having another helper function that
dumps a the size of the array to a file then get it via readData().
I'm just wondering if size-of-array queres or relative element
indexing is in the plans of implementation.

thanks,
-Allan


-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From benc at hawaga.org.uk  Sun Mar 21 17:55:48 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 21 Mar 2010 22:55:48 +0000 (GMT)
Subject: [Swift-user] last element array indexing?
In-Reply-To: <50b07b4b1003211521x7d111fd6rad1f7a931b0eef86@mail.gmail.com>
References: <50b07b4b1003211521x7d111fd6rad1f7a931b0eef86@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1003212253391.22136@dildano.hawaga.org.uk>


> How can I refer to the last element of the array?  my external mapper 
> doesn't know beforehand how many elements it will give the array.

Not at all answering your question but maybe giving some information that 
makes it even harder ;)

Although array indices must be integers (or perhaps numbers in general), 
there is no constraint that there are elements at any particular location. 
You can have an array containing values for f[3] and f[92]. So that has 2 
elements, and the last element has index 92. mmm.

I don't think there's much use for this 'feature', but there it is.

-- 
http://www.hawaga.org.uk/ben/



From benc at hawaga.org.uk  Mon Mar 22 06:32:12 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Mon, 22 Mar 2010 11:32:12 +0000 (GMT)
Subject: [Swift-user] last element array indexing?
In-Reply-To: <50b07b4b1003211521x7d111fd6rad1f7a931b0eef86@mail.gmail.com>
References: <50b07b4b1003211521x7d111fd6rad1f7a931b0eef86@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1003221120420.22136@dildano.hawaga.org.uk>


If you have map and fold, my beloved functional operators, you get to say 
(using haskell-like syntax, because swiftscript has no lambda or 
operators-as-functions).


 how many elements in a? = fold (+) (map (\x->1)) a

 whats the last element in a? = fold (\(l,r) -> r) a
 (though that doesn't return an index or anything lvalue like, so you can 
  only use this to *read* the last element in a, not write some last 
  value)

That first one is quite like the way counting is done in google 
map-reduce.

Actual working haskell code:

foldl1 (+) (map (\x->1) [543,213,765])
3

foldl1 (\l->(\r->r)) [1,2,3,4,5]
5


-- 
http://www.hawaga.org.uk/ben/



From aespinosa at cs.uchicago.edu  Thu Mar 25 15:28:49 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Thu, 25 Mar 2010 15:28:49 -0500
Subject: [Swift-user] mapping variables on the fly
Message-ID: <50b07b4b1003251328y4971c12eq71621ea650c12148@mail.gmail.com>

When my rups array contains more than one element, my script doesn't
seem to progress in executing.  the workdirectory for the extract()
job isn't even being prepared. My logfile says something like
"variable already closed".  its attached in this email.


below is the script, i swapped the "main" part of the program with the
type and app() declarations
/* TODO: data management zip jobs */
int run_id = 644;
Station site = get_site(run_id);

Sgt sgt_var <ext; exec="getsgtvar.rb", r=run_id, s=site.name,
    l="gsiftp://ff-grid.unl.edu//panfs/panasas/CMS/data/engage/scec/data/SgtFiles/TEST">;

Rupture rups[] = get_ruptures(run_id);

foreach rup in rups {
  string datadir =
      "gsiftp://ff-grid.unl.edu//panfs/panasas/CMS/data/engage/swift";
  string loc_sub = @strcat(datadir, ",", rup.index, "/", rup.source);
  Sgt sub <ext; exec="getsub.rb", l=loc_sub, n=site.name, s=rup.source,
      r=rup.index>;
  Variation vars[] <ext; exec="variation_mapper.rb", e=site.erf,
      v=site.variation_scenario, s=rup.source, r=rup.index>;

  sub = extract(sgt_var,  site, vars[rup.size-1]);

  foreach var,i in vars {
    Seismogram seis <single_file_mapper; file=@strcat(loc_sub, "/Seismogram_",
        site.name, "_", rup.source, "_", rup.index, "_", i,".grm")>;
    PeakValue peak <single_file_mapper; file=@strcat(loc_sub, "/PeakVals_",
        site.name, "_", rup.source, "_", rup.index, "_", i, ".bsa")>;

    seis = seismogram(sub, var, site);
    peak = peak_calc(seis, var);
  }
}


type SgtDim;
type Variation;
type Seismogram;
type PeakValue;

type Station {
  string name;
  float lat;
  float lon;
  int erf;
  int variation_scenario;
}

type Sgt {
  SgtDim x;
  SgtDim y;
}

type Rupture {
  int source;
  int index;
  int size;
}

/* some constants used by the apps*/
global int num_time_steps = 3000;
global string spectra_period1 = "all";
global float filter_highhz = 5.0;
global float simulation_timeskip = 0.1;

app (Sgt _ext) extract(Sgt _sgt, Station _stat, Variation _var) {
  jbsim3d @strcat("stat=", _stat.name) "extract_sgt=1"
      @strcat("slon=", _stat.lon) @strcat("slat=", _stat.lat)

      @strcat("rupmodfile=", @filename(_var))
      @strcat("sgt_xfile=", @filename(_sgt.x))
      @strcat("sgt_yfile=", @filename(_sgt.y))
      @strcat("extract_sgt_xfile=", @filename(_ext.x))
      @strcat("extract_sgt_yfile=", @filename(_ext.y));
}

app (Seismogram _seis) seismogram(Sgt _sgt, Variation _var, Station _stat) {
  jbsim3d @strcat("stat=", _stat.name) "extract_sgt=0"
      @strcat("slon=", _stat.lon) @strcat("slat=", _stat.lat)
      "outputBinary=1" "mergeOutput=1" @strcat("ntout=", num_time_steps)

      @strcat("rupmodfile=", @filename(_var))
      @strcat("sgt_xfile=", @filename(_sgt.x))
      @strcat("sgt_yfile=", @filename(_sgt.y))
      @strcat("seis_file=", @filename(_seis));
}

app (PeakValue _peak) peak_calc(Seismogram _seis, Variation _var) {
//  touch @_peak;
  surfeis_rspectra "simulation_out_pointsX=2" "simulation_out_pointsY=1"
      "surfseis_rspectra_seismogram_units=cmpersec"
      "surfseis_rspectra_output_units=cmpersec2"
      "surfseis_rspectra_output_type=aa"
      "surfseis_rspectra_apply_byteswap=no"

      @strcat("simulation_out_timesamples=", num_time_steps)
      @strcat("simulation_out_timeskip=", simulation_timeskip)
      @strcat("surfseis_rspectra_period=", spectra_period1)
      @strcat(" surfseis_rspectra_apply_filter_highHZ=", filter_highhz)
      @strcat("in=", @filename(_seis))
      @strcat("out=", @filename(_peak));
}

// Auxillary functions for the mappers
type StationFile;
app (StationFile _stat) getsite_file(int _run_id) {
  getsite _run_id stdout=@filename(_stat);
}
(Station _stat) get_site(int _run_id) {
  StationFile file<"site_tmp">;
  file = getsite_file(_run_id);
  _stat = readData(file);
}

type RuptureFile;
app (RuptureFile _rup) getrupture_file(int _run_id) {
  getrupture _run_id stdout=@filename(_rup);
}
(Rupture _rup[]) get_ruptures(int _run_id) {
  RuptureFile file<"rup_tmp">;
  file = getrupture_file(_run_id);
  _rup = readData(file);
}


--
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postproc-TEST.log
Type: text/x-log
Size: 175107 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20100325/fb40cbb4/attachment.bin>

From wozniak at mcs.anl.gov  Thu Mar 25 15:45:48 2010
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 25 Mar 2010 15:45:48 -0500 (Central Daylight Time)
Subject: [Swift-user] mapping variables on the fly
In-Reply-To: <50b07b4b1003251328y4971c12eq71621ea650c12148@mail.gmail.com>
References: <50b07b4b1003251328y4971c12eq71621ea650c12148@mail.gmail.com>
Message-ID: <alpine.WNT.2.00.1003251545390.3264@justinwozniak>


Is rups[] getting filled correctly?

On Thu, 25 Mar 2010, Allan Espinosa wrote:

> When my rups array contains more than one element, my script doesn't
> seem to progress in executing.  the workdirectory for the extract()
> job isn't even being prepared. My logfile says something like
> "variable already closed".  its attached in this email.
>
>
> below is the script, i swapped the "main" part of the program with the
> type and app() declarations
> /* TODO: data management zip jobs */
> int run_id = 644;
> Station site = get_site(run_id);
>
> Sgt sgt_var <ext; exec="getsgtvar.rb", r=run_id, s=site.name,
>    l="gsiftp://ff-grid.unl.edu//panfs/panasas/CMS/data/engage/scec/data/SgtFiles/TEST">;
>
> Rupture rups[] = get_ruptures(run_id);
>
> foreach rup in rups {
>  string datadir =
>      "gsiftp://ff-grid.unl.edu//panfs/panasas/CMS/data/engage/swift";
>  string loc_sub = @strcat(datadir, ",", rup.index, "/", rup.source);
>  Sgt sub <ext; exec="getsub.rb", l=loc_sub, n=site.name, s=rup.source,
>      r=rup.index>;
>  Variation vars[] <ext; exec="variation_mapper.rb", e=site.erf,
>      v=site.variation_scenario, s=rup.source, r=rup.index>;
>
>  sub = extract(sgt_var,  site, vars[rup.size-1]);
>
>  foreach var,i in vars {
>    Seismogram seis <single_file_mapper; file=@strcat(loc_sub, "/Seismogram_",
>        site.name, "_", rup.source, "_", rup.index, "_", i,".grm")>;
>    PeakValue peak <single_file_mapper; file=@strcat(loc_sub, "/PeakVals_",
>        site.name, "_", rup.source, "_", rup.index, "_", i, ".bsa")>;
>
>    seis = seismogram(sub, var, site);
>    peak = peak_calc(seis, var);
>  }
> }
>
>
> type SgtDim;
> type Variation;
> type Seismogram;
> type PeakValue;
>
> type Station {
>  string name;
>  float lat;
>  float lon;
>  int erf;
>  int variation_scenario;
> }
>
> type Sgt {
>  SgtDim x;
>  SgtDim y;
> }
>
> type Rupture {
>  int source;
>  int index;
>  int size;
> }
>
> /* some constants used by the apps*/
> global int num_time_steps = 3000;
> global string spectra_period1 = "all";
> global float filter_highhz = 5.0;
> global float simulation_timeskip = 0.1;
>
> app (Sgt _ext) extract(Sgt _sgt, Station _stat, Variation _var) {
>  jbsim3d @strcat("stat=", _stat.name) "extract_sgt=1"
>      @strcat("slon=", _stat.lon) @strcat("slat=", _stat.lat)
>
>      @strcat("rupmodfile=", @filename(_var))
>      @strcat("sgt_xfile=", @filename(_sgt.x))
>      @strcat("sgt_yfile=", @filename(_sgt.y))
>      @strcat("extract_sgt_xfile=", @filename(_ext.x))
>      @strcat("extract_sgt_yfile=", @filename(_ext.y));
> }
>
> app (Seismogram _seis) seismogram(Sgt _sgt, Variation _var, Station _stat) {
>  jbsim3d @strcat("stat=", _stat.name) "extract_sgt=0"
>      @strcat("slon=", _stat.lon) @strcat("slat=", _stat.lat)
>      "outputBinary=1" "mergeOutput=1" @strcat("ntout=", num_time_steps)
>
>      @strcat("rupmodfile=", @filename(_var))
>      @strcat("sgt_xfile=", @filename(_sgt.x))
>      @strcat("sgt_yfile=", @filename(_sgt.y))
>      @strcat("seis_file=", @filename(_seis));
> }
>
> app (PeakValue _peak) peak_calc(Seismogram _seis, Variation _var) {
> //  touch @_peak;
>  surfeis_rspectra "simulation_out_pointsX=2" "simulation_out_pointsY=1"
>      "surfseis_rspectra_seismogram_units=cmpersec"
>      "surfseis_rspectra_output_units=cmpersec2"
>      "surfseis_rspectra_output_type=aa"
>      "surfseis_rspectra_apply_byteswap=no"
>
>      @strcat("simulation_out_timesamples=", num_time_steps)
>      @strcat("simulation_out_timeskip=", simulation_timeskip)
>      @strcat("surfseis_rspectra_period=", spectra_period1)
>      @strcat(" surfseis_rspectra_apply_filter_highHZ=", filter_highhz)
>      @strcat("in=", @filename(_seis))
>      @strcat("out=", @filename(_peak));
> }
>
> // Auxillary functions for the mappers
> type StationFile;
> app (StationFile _stat) getsite_file(int _run_id) {
>  getsite _run_id stdout=@filename(_stat);
> }
> (Station _stat) get_site(int _run_id) {
>  StationFile file<"site_tmp">;
>  file = getsite_file(_run_id);
>  _stat = readData(file);
> }
>
> type RuptureFile;
> app (RuptureFile _rup) getrupture_file(int _run_id) {
>  getrupture _run_id stdout=@filename(_rup);
> }
> (Rupture _rup[]) get_ruptures(int _run_id) {
>  RuptureFile file<"rup_tmp">;
>  file = getrupture_file(_run_id);
>  _rup = readData(file);
> }
>
>
> --
> Allan M. Espinosa <http://amespinosa.wordpress.com>
> PhD student, Computer Science
> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
>

-- 
Justin M Wozniak


From fedorov at bwh.harvard.edu  Sat Mar 27 15:58:36 2010
From: fedorov at bwh.harvard.edu (Andriy Fedorov)
Date: Sat, 27 Mar 2010 16:58:36 -0400
Subject: [Swift-user] @tostring?
Message-ID: <82f536811003271358r7eb5fc4q6aef6837c3d16726@mail.gmail.com>

Hi,

Is it possible to convert int type to string type in Swift?

I see @toint, but not @tostring, and it looks like I am not able to
pass int to @strcat().

Thanks

--
Andriy Fedorov, Ph.D.

Research Fellow
Brigham and Women's Hospital
Harvard Medical School
75 Francis Street
Boston, MA 02115 USA
fedorov at bwh.harvard.edu


From wilde at mcs.anl.gov  Sat Mar 27 16:34:05 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Sat, 27 Mar 2010 16:34:05 -0500 (CDT)
Subject: [Swift-user] @tostring?
In-Reply-To: <82f536811003271358r7eb5fc4q6aef6837c3d16726@mail.gmail.com>
Message-ID: <27153021.381171269725645789.JavaMail.root@zimbra>

Andriy, Im pretty sure @strcat() will take an int and return a string.

- Mike

----- "Andriy Fedorov" <fedorov at bwh.harvard.edu> wrote:

> Hi,
> 
> Is it possible to convert int type to string type in Swift?
> 
> I see @toint, but not @tostring, and it looks like I am not able to
> pass int to @strcat().
> 
> Thanks
> 
> --
> Andriy Fedorov, Ph.D.
> 
> Research Fellow
> Brigham and Women's Hospital
> Harvard Medical School
> 75 Francis Street
> Boston, MA 02115 USA
> fedorov at bwh.harvard.edu
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From fedorov at bwh.harvard.edu  Sat Mar 27 16:38:11 2010
From: fedorov at bwh.harvard.edu (Andriy Fedorov)
Date: Sat, 27 Mar 2010 17:38:11 -0400
Subject: [Swift-user] @tostring?
In-Reply-To: <27153021.381171269725645789.JavaMail.root@zimbra>
References: <82f536811003271358r7eb5fc4q6aef6837c3d16726@mail.gmail.com>
	<27153021.381171269725645789.JavaMail.root@zimbra>
Message-ID: <82f536811003271438i5ee74ce5k83c661a5dca3a5b8@mail.gmail.com>

On Sat, Mar 27, 2010 at 17:34, Michael Wilde <wilde at mcs.anl.gov> wrote:
> Andriy, Im pretty sure @strcat() will take an int and return a string.
>

Mike, yes, you are right -- there was another error in my script,
@strcat indeed works with int.

One more basic question (sorry if I missed this in the guide): is
there a way to get the size of an array? I would like to split a
string, and get the last item in the array it returns. Is this
possible?


> - Mike
>
> ----- "Andriy Fedorov" <fedorov at bwh.harvard.edu> wrote:
>
>> Hi,
>>
>> Is it possible to convert int type to string type in Swift?
>>
>> I see @toint, but not @tostring, and it looks like I am not able to
>> pass int to @strcat().
>>
>> Thanks
>>
>> --
>> Andriy Fedorov, Ph.D.
>>
>> Research Fellow
>> Brigham and Women's Hospital
>> Harvard Medical School
>> 75 Francis Street
>> Boston, MA 02115 USA
>> fedorov at bwh.harvard.edu
>> _______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>


From wilde at mcs.anl.gov  Sat Mar 27 17:10:49 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Sat, 27 Mar 2010 17:10:49 -0500 (CDT)
Subject: [Swift-user] @tostring?
In-Reply-To: <28638685.381601269727528186.JavaMail.root@zimbra>
Message-ID: <9408326.381681269727849710.JavaMail.root@zimbra>

Andriy,

At the moment (and maybe forever ;) "sizeof()" is a black hole. Dont go there if possible.

It depends somewhat on what you want to do with the rest of the string, but if all you want to do is process the last segment of the string, what I would do is add tiny external functions that execute on localhost to deal with things like that (I.e, just grabbing the last segment of a string).

I can provide 3 tips for doing this.

(0) note that @strcat() will also convert floats, and I presume, boolean.

(1) I started to play with using "tiny" fast external apps like echo, sed, awk to start building up a useful library of string functions. I didnt take this any further than what I needed at the time; but I believe the technique is useful.

I paste below my crude start at this. With a few entries for filters on localhost, you can do quite a bit. Ignore performance (its usually fine); think of it like doing command interpolation in the shell (`` or $(cmd) )

(2) With a single entry for a "swiftshell" in tc.data, you can compose shell filters in Swift itself. I paste that below too. Beware, though - Sarah seems to have encountered a bug or issue in which a provider (likely coasters) is wiping out a "$1" in one such shell. But I think localhost is OK, which is what youd want to use.

(3) Justin has added an @java() primitive to the development trunk; so if there's a Java method among standard or Swift Java classes that you can call to do a needed string operation, that may be useful. I cant recall if that was posted to one of the lists or not.

Hope one of these works for you; I think (2) is a pretty good starting bet. At least one Google GSoC student has indicated interest in building up a starter Swift library for the community, so please post your needs to this list as they come up.

- Mike

=== (1) - A "toy" Swift library:

// General Swift Lib Functions

app (file o) echoi (int i)     { echo i stdout=@o;}
app (file o) echof (float f)   { echo f stdout=@o;}
app (file o) echob (boolean b) { echo b stdout=@o;}
app (file o) echos (string s)  { echo s stdout=@o;}

(string s) itostr (int i)
{
  file f;
  f = echoi(i);
  s = readData(f);
}

(string s) ftostr (float n)
{
  file f;
  f = echof(n);
  s = readData(f);
}

(int n) strtoi (string s)
{
  file f;
  f = echos(s);
  n = readData(f);
}

(float n) strtof (string s)
{
  file f;
  f = echos(s);
  n = readData(f);
}

app (file o) sprintfsApp (string fmt, string e[])
{
  sprintfs fmt e stdout=@o;
}

(string s) sprintfs (string fmt, string e[])
{
  file f;
  f = sprintfsApp(fmt,e);
  s = readData(f);
}

=== (2) swiftshell:

login1$ more shelldemo.swift swiftshell
::::::::::::::
shelldemo.swift
::::::::::::::
type file;

app (file o) cat (file i)
{
  shell " ( cat " @i "; date; hostname ) | grep . " stdout=@o;
}

file data<"data.txt">;
file out<"out.txt">;
out = cat(data);
::::::::::::::
swiftshell
::::::::::::::
bash -c "$*"
login1$ grep shell tc
localhost 	shell	/home/wilde/swift/lab/swiftshell	null	null	null
mcs	 	shell	/home/wilde/swift/lab/swiftshell	null	null	null
login1$ 

=== (3) @java()

Justin previously posted this to the list:

"         If you can check out the latest Swift from trunk I've added some 
features that might help you out here.  There's a new built-in function @java() 
that allows you to call into an existing Java library.  You can call into the 
Java Platform or into your CLASSPATH.
         Here is one example:

(float result) sin(float x) {
   result = @java("java.lang.Math", "sin", x);
}

float x = 0.5;
float y = sin(x);

trace("sin", x, y);

Note that you currently have to assign the result of @java() to a variable."




----- "Andriy Fedorov" <fedorov at bwh.harvard.edu> wrote:

> On Sat, Mar 27, 2010 at 17:34, Michael Wilde <wilde at mcs.anl.gov>
> wrote:
> > Andriy, Im pretty sure @strcat() will take an int and return a
> string.
> >
> 
> Mike, yes, you are right -- there was another error in my script,
> @strcat indeed works with int.
> 
> One more basic question (sorry if I missed this in the guide): is
> there a way to get the size of an array? I would like to split a
> string, and get the last item in the array it returns. Is this
> possible?
> 
> 
> > - Mike
> >
> > ----- "Andriy Fedorov" <fedorov at bwh.harvard.edu> wrote:
> >
> >> Hi,
> >>
> >> Is it possible to convert int type to string type in Swift?
> >>
> >> I see @toint, but not @tostring, and it looks like I am not able
> to
> >> pass int to @strcat().
> >>
> >> Thanks
> >>
> >> --
> >> Andriy Fedorov, Ph.D.
> >>
> >> Research Fellow
> >> Brigham and Women's Hospital
> >> Harvard Medical School
> >> 75 Francis Street
> >> Boston, MA 02115 USA
> >> fedorov at bwh.harvard.edu
> >> _______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From aespinosa at cs.uchicago.edu  Sat Mar 27 20:45:31 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Sat, 27 Mar 2010 19:45:31 -0600
Subject: [Swift-user] Swiftscript vim syntax file
Message-ID: <50b07b4b1003271845g2730b924p3aa22163a225d6a7@mail.gmail.com>

Just a quick weekend hack.

Installation:

1. get swift.vim from http://www.ci.uchicago.edu/~aespinosa/swift/swift.vim
2. place swift.vim in your ~/.vim/syntax directory
3. add the following lines to your ~/.vim/filetype.vim :
" Swift script
au BufRead,BufNewFile *.swift setfiletype swift

Now opening a .swift file in your ViM text editor will now have pretty colors :)

Enjoy!
-Allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From benc at hawaga.org.uk  Sun Mar 28 04:43:30 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 28 Mar 2010 09:43:30 +0000 (GMT)
Subject: [Swift-user] Swiftscript vim syntax file
In-Reply-To: <50b07b4b1003271845g2730b924p3aa22163a225d6a7@mail.gmail.com>
References: <50b07b4b1003271845g2730b924p3aa22163a225d6a7@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1003280942510.7388@dildano.hawaga.org.uk>


I often thought about making one of those, but always got side tracked 
into wondering how to generate one automatically from the existing antlr 
syntax definitions ;)

-- 



From foster at anl.gov  Sat Mar 27 21:57:48 2010
From: foster at anl.gov (Ian Foster)
Date: Sat, 27 Mar 2010 21:57:48 -0500
Subject: [Swift-user] Swiftscript vim syntax file
In-Reply-To: <50b07b4b1003271845g2730b924p3aa22163a225d6a7@mail.gmail.com>
References: <50b07b4b1003271845g2730b924p3aa22163a225d6a7@mail.gmail.com>
Message-ID: <B1734762-5A7D-41C0-8872-2DF6B2EAEBE8@anl.gov>

let's make sure this is on the Swift web somewhere!

On Mar 27, 2010, at 8:45 PM, Allan Espinosa wrote:

> Just a quick weekend hack.
> 
> Installation:
> 
> 1. get swift.vim from http://www.ci.uchicago.edu/~aespinosa/swift/swift.vim
> 2. place swift.vim in your ~/.vim/syntax directory
> 3. add the following lines to your ~/.vim/filetype.vim :
> " Swift script
> au BufRead,BufNewFile *.swift setfiletype swift
> 
> Now opening a .swift file in your ViM text editor will now have pretty colors :)
> 
> Enjoy!
> -Allan
> 
> -- 
> Allan M. Espinosa <http://amespinosa.wordpress.com>
> PhD student, Computer Science
> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user



From yizhu at cs.uchicago.edu  Mon Mar 29 17:31:48 2010
From: yizhu at cs.uchicago.edu (Yi Zhu)
Date: Mon, 29 Mar 2010 17:31:48 -0500
Subject: [Swift-user] How to add resources constraint in sites.xml for qsub
Message-ID: <4BB12A54.3010104@cs.uchicago.edu>

Hi everyone,

we've already known that we can describe the queue type in sites.xml:

e.g:

qsub -q short myjobs.pbs

can be written in sites.xml by  the following statement:

<profile namespace="globus" key="queue">short</profile>


But what if I want to specify some other resources constraint by using 
"-l argument":

e.g:

qsub -l mem=3gb myjobs.pbs

How can I describe that in sites.xml files?



Many Thanks!

-Yi Zhu