From aespinosa at cs.uchicago.edu  Wed Feb  3 15:38:01 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 3 Feb 2010 15:38:01 -0600
Subject: [Swift-devel] GLOBUS::count gives invalid RSL requests
Message-ID: <50b07b4b1002031338n25155dc3x5aaaf590e959c4f3@mail.gmail.com>

Hi I was trying to use profile namespaces on tc.data

RANGER  mpiblast
/work/01035/tg802895/science/jgiblast/wrapper_mpiblast.sh INSTALLED
INTEL32::LINUX
GLOBUS::count="1024";GLOBUS::host_count="16";GLOBUS::maxwalltime="1:00:00";GLOBUS::job_type="single"

and gives me these error message of generating an incorrect RSL script:
 svn swift-r3202 cog-r2683

RunID: testing_ranger
Progress:
Progress:  Selecting site:1  Stage in:2
Progress:  Selecting site:1  Submitting:1  Submitted:1
Progress:  Selecting site:1  Submitted:1  Active:1
Failed to transfer wrapper log from mpiblastp-testing_ranger/info/h on RANGER
Progress:  Selecting site:1  Failed:1 Failed but can retry:1
Execution failed:
	Exception in mpiblast:
Arguments: [--replica-group-size, 32, --use-parallel-write,
--partition-size, 257, --use-virtual-frags, -p, blastp, -m, 8, -e,
0.001, -F, F, -d, nr6_00/nr00, -i,
/work/01035/tg802895/nr_andreas/reciprocal/reciprocal/nr00-0, -o,
out/res_nr00-0.out]
Host: RANGER
Directory: mpiblastp-testing_ranger/jobs/h/mpiblast-hmbbm9nj
stderr.txt:

stdout.txt:

----

Caused by:
	The provided RSL 'count' value is invalid (not an integer or must be
greater than 0)


Putting this on <profile namespace="globus" key="count">1024</profile>
works correctly in my workflow.

thanks,
-allan

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>


From aespinosa at cs.uchicago.edu  Wed Feb  3 15:41:08 2010
From: aespinosa at cs.uchicago.edu (Allan Espinosa)
Date: Wed, 3 Feb 2010 15:41:08 -0600
Subject: [Swift-devel] Re: GLOBUS::count gives invalid RSL requests
In-Reply-To: <50b07b4b1002031338n25155dc3x5aaaf590e959c4f3@mail.gmail.com>
References: <50b07b4b1002031338n25155dc3x5aaaf590e959c4f3@mail.gmail.com>
Message-ID: <50b07b4b1002031341y47c79c5j4cbc2236b7677190@mail.gmail.com>

Ooops. nevermind.  I just missed specifying my queue that allows me
larger cpu counts.



2010/2/3 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> Hi I was trying to use profile namespaces on tc.data
>
> RANGER ?mpiblast
> /work/01035/tg802895/science/jgiblast/wrapper_mpiblast.sh INSTALLED
> INTEL32::LINUX
> GLOBUS::count="1024";GLOBUS::host_count="16";GLOBUS::maxwalltime="1:00:00";GLOBUS::job_type="single"
>
> and gives me these error message of generating an incorrect RSL script:
> ?svn swift-r3202 cog-r2683
>
> RunID: testing_ranger
> Progress:
> Progress: ?Selecting site:1 ?Stage in:2
> Progress: ?Selecting site:1 ?Submitting:1 ?Submitted:1
> Progress: ?Selecting site:1 ?Submitted:1 ?Active:1
> Failed to transfer wrapper log from mpiblastp-testing_ranger/info/h on RANGER
> Progress: ?Selecting site:1 ?Failed:1 Failed but can retry:1
> Execution failed:
> ? ? ? ?Exception in mpiblast:
> Arguments: [--replica-group-size, 32, --use-parallel-write,
> --partition-size, 257, --use-virtual-frags, -p, blastp, -m, 8, -e,
> 0.001, -F, F, -d, nr6_00/nr00, -i,
> /work/01035/tg802895/nr_andreas/reciprocal/reciprocal/nr00-0, -o,
> out/res_nr00-0.out]
> Host: RANGER
> Directory: mpiblastp-testing_ranger/jobs/h/mpiblast-hmbbm9nj
> stderr.txt:
>
> stdout.txt:
>
> ----
>
> Caused by:
> ? ? ? ?The provided RSL 'count' value is invalid (not an integer or must be
> greater than 0)
>
>
> Putting this on <profile namespace="globus" key="count">1024</profile>
> works correctly in my workflow.
>
> thanks,
> -allan
>


From foster at anl.gov  Sun Feb  7 14:32:51 2010
From: foster at anl.gov (Ian Foster)
Date: Sun, 7 Feb 2010 14:32:51 -0600
Subject: [Swift-devel] Fwd: [Swift-user] Specifying execution dependencies
	directly
References: <1265482121.28632.2.camel@localhost>
Message-ID: <2D53FA99-8C15-40E5-B7C2-ED32B88CC504@anl.gov>

This would be a nice thing to have in the Swift manual: "how to do mapreduce problems"

Begin forwarded message:

> From: Mihael Hategan <hategan at mcs.anl.gov>
> Date: February 6, 2010 12:48:41 PM CST
> To: Andriy Fedorov <fedorov at bwh.harvard.edu>
> Cc: swift-user at ci.uchicago.edu
> Subject: Re: [Swift-user] Specifying execution dependencies directly
> 
> On Sat, 2010-02-06 at 13:03 -0500, Andriy Fedorov wrote:
>> Hi,
>> 
>> I have the following (what seems to me, typical) analysis scenario. I
>> am running multiple instances of task A, task B needs to wait until
>> all of the instances of A are completed, and analyze the outputs
>> produced by As.
>> 
>> If I was running this locally, I would pass the name of the directory
>> with the results to B. However, in Swift, I need to somehow define
>> that B cannot start until all of the As complete. Since B would expect
>> a directory name, or an arbitrarily long list of files, I am not sure
>> how this can be done. Should I develop the workflow as a shell script
>> that would run swift script with As, then run B locally, then proceed
>> with the rest of the computation?
> 
> You are right, this is a typical map/reduce problem.
> 
> Use an array that holds the results of all the As and pass that to B.
> 
> file aout[];
> foreach v, k in someRangeOrSomething {
>  aout[k] = a(v);
> }
> file bout;
> bout = b(aout);
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100207/16d85648/attachment.html>

From iraicu at cs.uchicago.edu  Mon Feb  8 18:25:05 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Mon, 08 Feb 2010 18:25:05 -0600
Subject: [Swift-devel] CFP: ACM ScienceCloud 2010
Message-ID: <4B70AB61.7070308@cs.uchicago.edu>

CFP reminder: ACM ScienceCloud 2010

This is a kind reminder that the submission
deadline for ScienceCloud 2010 is approaching:

Event Name: ACM Workshop on Scientific Cloud Computing
                         (ScienceCloud) 2010

Web site:   http://dsl.cs.uchicago.edu/ScienceCloud2010/

Venue:      High Performance Distributed Computing
                         Symposium (HPDC 2010)

Date:       June 21st, 2010

Location:   Chicago, Illinois, USA

Deadlines:
            Abstract:   February 22nd, 2010
            Full paper: March 1st, 2010
            Acceptance Notification: April 1st, 2010
            Camera Ready Papers:  April 15th, 2010


-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================



-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




From iraicu at cs.uchicago.edu  Tue Feb  9 17:07:31 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Tue, 09 Feb 2010 17:07:31 -0600
Subject: [Swift-devel] DAG visualization tool
Message-ID: <4B71EAB3.8010904@cs.uchicago.edu>

Hi,
Can anyone point me to the tool used to visualize DAGs, like the one I 
found in one of the Swift papers? Also, does the tool require 
SwiftScript to generate the graph, or does it only consume a DAG 
description, and therefore could be used outside of Swift as well, as 
long as you describe the DAG in the right format?

Thanks,
Ioan

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100209/ce168dad/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moz-screenshot-1.jpg
Type: image/jpeg
Size: 120387 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100209/ce168dad/attachment.jpg>

From wilde at mcs.anl.gov  Tue Feb  9 17:21:20 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 9 Feb 2010 17:21:20 -0600 (CST)
Subject: [Swift-devel] DAG visualization tool
In-Reply-To: <4B71EAB3.8010904@cs.uchicago.edu>
Message-ID: <24495828.432001265757680634.JavaMail.root@zimbra>


----- "Ioan Raicu" <iraicu at cs.uchicago.edu> wrote:

> Hi,
> Can anyone point me to the tool used to visualize DAGs, like the one I
> found in one of the Swift papers?

Ioan, its the "pgraph" options that can be set in the propoerties file of command line:

http://www.ci.uchicago.edu/swift/guides/userguide.php#engineconfiguration

> Also, does the tool require
> SwiftScript to generate the graph, or does it only consume a DAG
> description, and therefore could be used outside of Swift as well, as
> long as you describe the DAG in the right format?

I think it requires Swift to gen the graph from a Swift source file, as there is no other DAG description involved (beyond the generated Karajan representation).

- Mike

> 
> Thanks,
> Ioan
> 
> --
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu Web:
> http://www.eecs.northwestern.edu/~iraicu/
> https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> ================================================================= 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From skenny at uchicago.edu  Tue Feb  9 18:13:10 2010
From: skenny at uchicago.edu (skenny at uchicago.edu)
Date: Tue,  9 Feb 2010 18:13:10 -0600 (CST)
Subject: [Swift-devel] how to comment out brackets
Message-ID: <20100209181310.CJM44697@m4500-02.uchicago.edu>

does anyone know if there's a way i can comment out a bracket
in a swift string?

for example something like this:

string cmdstr = "\{";
trace(cmdstr);

produces:

Could not compile SwiftScript source: line 15:27: unexpected
char: '{'

not sure if it's a bug or i'm just missing something really
simple here :)

~sk


From hategan at mcs.anl.gov  Tue Feb  9 19:31:05 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 09 Feb 2010 19:31:05 -0600
Subject: [Swift-devel] how to comment out brackets
In-Reply-To: <20100209181310.CJM44697@m4500-02.uchicago.edu>
References: <20100209181310.CJM44697@m4500-02.uchicago.edu>
Message-ID: <1265765465.3915.2.camel@localhost>

On Tue, 2010-02-09 at 18:13 -0600, skenny at uchicago.edu wrote:
> does anyone know if there's a way i can comment out a bracket
> in a swift string?
> 
> for example something like this:
> 
> string cmdstr = "\{";
> trace(cmdstr);
> 
> produces:
> 
> Could not compile SwiftScript source: line 15:27: unexpected
> char: '{'
> 
> not sure if it's a bug or i'm just missing something really
> simple here :)

You shouldn't need to escape them (and the swift parser probably
complains because it doesn't recognize the escape sequence), but I have
a suspicion you're sending this email because you had a problem, and I
suspect that problem is that the swift compiler passes brackets
unescaped to karajan, in which the bracket means something.

If that's the case, try "{{" to generate one bracket. You don't need to
escape the closing bracket (i.e. "}").



From skenny at uchicago.edu  Tue Feb  9 20:32:28 2010
From: skenny at uchicago.edu (skenny at uchicago.edu)
Date: Tue,  9 Feb 2010 20:32:28 -0600 (CST)
Subject: [Swift-devel] how to comment out brackets
In-Reply-To: <1265765465.3915.2.camel@localhost>
References: <20100209181310.CJM44697@m4500-02.uchicago.edu>
	<1265765465.3915.2.camel@localhost>
Message-ID: <20100209203228.CJM56455@m4500-02.uchicago.edu>

>On Tue, 2010-02-09 at 18:13 -0600, skenny at uchicago.edu wrote:
>> does anyone know if there's a way i can comment out a bracket
>> in a swift string?
>> 
>> for example something like this:
>> 
>> string cmdstr = "\{";
>> trace(cmdstr);
>> 
>> produces:
>> 
>> Could not compile SwiftScript source: line 15:27: unexpected
>> char: '{'
>> 
>> not sure if it's a bug or i'm just missing something really
>> simple here :)
>
>You shouldn't need to escape them (and the swift parser probably
>complains because it doesn't recognize the escape sequence),
but I have
>a suspicion you're sending this email because you had a
problem, and I
>suspect that problem is that the swift compiler passes brackets
>unescaped to karajan, in which the bracket means something.
>
>If that's the case, try "{{" to generate one bracket. You
don't need to
>escape the closing bracket (i.e. "}").

that did it, thanks! double open-bracket gets it thru (yeah, i
had tried not escaping it at all and needless to say it
produced an error as well). 




From benc at hawaga.org.uk  Wed Feb 10 03:49:35 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 10 Feb 2010 09:49:35 +0000 (GMT)
Subject: [Swift-devel] DAG visualization tool
In-Reply-To: <4B71EAB3.8010904@cs.uchicago.edu>
References: <4B71EAB3.8010904@cs.uchicago.edu>
Message-ID: <Pine.LNX.4.64.1002100752010.22136@dildano.hawaga.org.uk>

> Can anyone point me to the tool used to visualize DAGs, like the one I found
> in one of the Swift papers? Also, does the tool require SwiftScript to
> generate the graph, or does it only consume a DAG description, and therefore
> could be used outside of Swift as well, as long as you describe the DAG in the
> right format?

probably the dot command from graphviz.

The commandline options that mike gave for swift generate input for that, 
but graphviz in general is for any DAGs.

Its a pretty straightforward syntax to get basic graphs.

-- 



From iraicu at cs.uchicago.edu  Wed Feb 10 10:13:50 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Wed, 10 Feb 2010 10:13:50 -0600
Subject: [Swift-devel] DAG visualization tool
In-Reply-To: <Pine.LNX.4.64.1002100752010.22136@dildano.hawaga.org.uk>
References: <4B71EAB3.8010904@cs.uchicago.edu>
	<Pine.LNX.4.64.1002100752010.22136@dildano.hawaga.org.uk>
Message-ID: <4B72DB3E.3030307@cs.uchicago.edu>

Thanks Ben, I'll look into graphviz.

Cheers,
Ioan

Ben Clifford wrote:
>> Can anyone point me to the tool used to visualize DAGs, like the one I found
>> in one of the Swift papers? Also, does the tool require SwiftScript to
>> generate the graph, or does it only consume a DAG description, and therefore
>> could be used outside of Swift as well, as long as you describe the DAG in the
>> right format?
>>     
>
> probably the dot command from graphviz.
>
> The commandline options that mike gave for swift generate input for that, 
> but graphviz in general is for any DAGs.
>
> Its a pretty straightforward syntax to get basic graphs.
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100210/8cff2cdd/attachment.html>

From mmaiensc at uchicago.edu  Fri Feb 12 20:19:21 2010
From: mmaiensc at uchicago.edu (Mark Maienschein-Cline)
Date: Fri, 12 Feb 2010 20:19:21 -0600
Subject: [Swift-devel] Re: [Swift-user] Error using programs in swift
In-Reply-To: <30064923.551521266018474313.JavaMail.root@zimbra>
References: <30064923.551521266018474313.JavaMail.root@zimbra>
Message-ID: <9E49AB89-F96F-4546-B7B6-EB69AE33F83E@uchicago.edu>

Hi,
Here is the log file and the Swift script for the java  
NullPointerError I have been getting. Le me know if you also need the  
code for the application. I am running swift locally on the cluster  
sisboombah.uchicago.edu.
Thanks,
Mark

-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-20100212-2114-10exxr8d.log
Type: application/octet-stream
Size: 17304 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100212/094ad2bd/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.swift
Type: application/octet-stream
Size: 434 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100212/094ad2bd/attachment-0001.obj>
-------------- next part --------------



On Feb 12, 2010, at 5:47 PM, Michael Wilde wrote:

> Mark, a null ptr exception is always an internal Swift error  
> triggered by some problem in Java.
>
> Please send to swift-devel#ci.uchicago.edu the Swift script and the  
> Swift log file from the run (the one called some-long-name.log) -  
> or tell us where we can find it on the CI network.
>
> Thanks,
>
> Mike
>
>
> ----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:
>
>> Hi,
>>
>> I'm getting a "java.lang.NullPointerException" error when I call some
>> of my programs (written in C) in Swift (in the local mode). The
>> programs work correctly when I run them from the command line, and I
>> don't think there is any error in the parameters swift is passing to
>> the program. Does anyone know what problems can cause this kind of
>> error?
>>
>> Thanks,
>> Mark_______________________________________________
>> Swift-user mailing list
>> Swift-user at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wilde at mcs.anl.gov  Mon Feb 15 22:19:23 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Mon, 15 Feb 2010 22:19:23 -0600 (CST)
Subject: [Swift-devel] Re: [Swift-user] Error using programs in swift
In-Reply-To: <33224725.602711266293732847.JavaMail.root@zimbra>
Message-ID: <19971172.602761266293963182.JavaMail.root@zimbra>

Mark, the problem looks to me like Swift is not handing the following programming error correctly.

In the line:

   subpeaks @file1 @file2 @max_dist stdout=@output;

you dont want to put the @ on the int max_dist.

Remember, @ is shorthand for the @filename(), and you can't apply that to an int.

I *think* Swift should generate a compile-time type violation, but thats perhaps not do-able at the moment for @func() primitives, which have some type flexibility.

So the short answer is: take off the @ on max_dist. When you list a scalar variable in an app() command-line prototype, the scalar's value is placed on the command line.

- Mike
 


----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:

> Hi,
> Here is the log file and the Swift script for the java  
> NullPointerError I have been getting. Le me know if you also need the 
> 
> code for the application. I am running swift locally on the cluster  
> sisboombah.uchicago.edu.
> Thanks,
> Mark
> 
> 
> 
> 
> 
> On Feb 12, 2010, at 5:47 PM, Michael Wilde wrote:
> 
> > Mark, a null ptr exception is always an internal Swift error  
> > triggered by some problem in Java.
> >
> > Please send to swift-devel#ci.uchicago.edu the Swift script and the 
> 
> > Swift log file from the run (the one called some-long-name.log) -  
> > or tell us where we can find it on the CI network.
> >
> > Thanks,
> >
> > Mike
> >
> >
> > ----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:
> >
> >> Hi,
> >>
> >> I'm getting a "java.lang.NullPointerException" error when I call
> some
> >> of my programs (written in C) in Swift (in the local mode). The
> >> programs work correctly when I run them from the command line, and
> I
> >> don't think there is any error in the parameters swift is passing
> to
> >> the program. Does anyone know what problems can cause this kind of
> >> error?
> >>
> >> Thanks,
> >> Mark_______________________________________________
> >> Swift-user mailing list
> >> Swift-user at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wilde at mcs.anl.gov  Mon Feb 15 22:32:27 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Mon, 15 Feb 2010 22:32:27 -0600 (CST)
Subject: [Swift-devel] Re: [Swift-user] Error using programs in swift
In-Reply-To: <19971172.602761266293963182.JavaMail.root@zimbra>
Message-ID: <30444183.602961266294747429.JavaMail.root@zimbra>

The Ex098 error followed by an NPE can be reproduced with this simple program:

app f(int j)
{
 echo @j;
}

f(99);

- Mike

----- wilde at mcs.anl.gov wrote:

> Mark, the problem looks to me like Swift is not handing the following
> programming error correctly.
> 
> In the line:
> 
>    subpeaks @file1 @file2 @max_dist stdout=@output;
> 
> you dont want to put the @ on the int max_dist.
> 
> Remember, @ is shorthand for the @filename(), and you can't apply that
> to an int.
> 
> I *think* Swift should generate a compile-time type violation, but
> thats perhaps not do-able at the moment for @func() primitives, which
> have some type flexibility.
> 
> So the short answer is: take off the @ on max_dist. When you list a
> scalar variable in an app() command-line prototype, the scalar's value
> is placed on the command line.
> 
> - Mike
>  
> 
> 
> ----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:
> 
> > Hi,
> > Here is the log file and the Swift script for the java  
> > NullPointerError I have been getting. Le me know if you also need
> the 
> > 
> > code for the application. I am running swift locally on the cluster 
> 
> > sisboombah.uchicago.edu.
> > Thanks,
> > Mark
> > 
> > 
> > 
> > 
> > 
> > On Feb 12, 2010, at 5:47 PM, Michael Wilde wrote:
> > 
> > > Mark, a null ptr exception is always an internal Swift error  
> > > triggered by some problem in Java.
> > >
> > > Please send to swift-devel#ci.uchicago.edu the Swift script and
> the 
> > 
> > > Swift log file from the run (the one called some-long-name.log) - 
> 
> > > or tell us where we can find it on the CI network.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > >
> > > ----- "Mark Maienschein-Cline" <mmaiensc at uchicago.edu> wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm getting a "java.lang.NullPointerException" error when I call
> > some
> > >> of my programs (written in C) in Swift (in the local mode). The
> > >> programs work correctly when I run them from the command line,
> and
> > I
> > >> don't think there is any error in the parameters swift is
> passing
> > to
> > >> the program. Does anyone know what problems can cause this kind
> of
> > >> error?
> > >>
> > >> Thanks,
> > >> Mark_______________________________________________
> > >> Swift-user mailing list
> > >> Swift-user at ci.uchicago.edu
> > >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From benc at hawaga.org.uk  Tue Feb 16 05:11:19 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 16 Feb 2010 11:11:19 +0000 (GMT)
Subject: [Swift-devel] Re: [Swift-user] Error using programs in swift
In-Reply-To: <19971172.602761266293963182.JavaMail.root@zimbra>
References: <19971172.602761266293963182.JavaMail.root@zimbra>
Message-ID: <Pine.LNX.4.64.1002161110080.22136@dildano.hawaga.org.uk>

> I *think* Swift should generate a compile-time type violation, but thats 
> perhaps not do-able at the moment for @func() primitives, which have 
> some type flexibility.

It probably should do so now - but at some point in the past, there was a 
lack of consensus about whether types that have an in-memory value (such 
as int) could be mapped too (maybe there still is).

-- 



From tiberius at ci.uchicago.edu  Wed Feb 17 10:48:38 2010
From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun)
Date: Wed, 17 Feb 2010 10:48:38 -0600
Subject: [Swift-devel] Problem with iterate
Message-ID: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>

Hi Guys

I have put together this really simple iteration example, which I expected
to work (and it used to work in the past):

==============
type file;

app (file initOut) initFunc (string inputString){
        echo inputString stdout=@filename(initOut);
}

app (file catOut) catFunc (file catIn){
        cat @filename(catIn) @filename(catOut);
}

runLoop(){

        file statePathFiles[]<concurrent_mapper; prefix="statePathName.",
suffix=".mat">;

        statePathFiles[0]=initFunc("hello");

        iterate it{
                trace(@strcat("Iteration: ",it));
                statePathFiles[it+1]=catFunc(statePathFiles[it]);
        } until (it==0);
}

runLoop();
================

However, I get an error like this:

Running UC Eval
Could not start execution.
    variable statePathFiles has multiple writers.


Please suggest  solutions for it .

Thank you
Tibi



-- 
Tiberiu (Tibi) Stef-Praun, PhD
Computational Sciences Researcher
Computation Institute
5640 S. Ellis Ave, #405
University of Chicago
http://www-unix.mcs.anl.gov/~tiberius/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100217/aeb88544/attachment.html>

From wilde at mcs.anl.gov  Wed Feb 17 10:56:33 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 17 Feb 2010 10:56:33 -0600 (CST)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
Message-ID: <21055881.656131266425793594.JavaMail.root@zimbra>

Tibi, while it may not be the most elegant approach, I think you need to put all statements that set the array statePathFiles *inside* the loop, and conditionally execute the setting of element 0 using an if statement.

- Mike


----- "Tiberiu Stef-Praun" <tiberius at ci.uchicago.edu> wrote:

> Hi Guys
> 
> I have put together this really simple iteration example, which I
> expected to work (and it used to work in the past):
> 
> ==============
> type file;
> 
> app (file initOut) initFunc (string inputString){
> echo inputString stdout=@filename(initOut);
> }
> 
> app (file catOut) catFunc (file catIn){
> cat @filename(catIn) @filename(catOut);
> }
> 
> runLoop(){
> 
> file statePathFiles[]<concurrent_mapper; prefix="statePathName.",
> suffix=".mat">;
> 
> statePathFiles[0]=initFunc("hello");
> 
> iterate it{
> trace(@strcat("Iteration: ",it));
> statePathFiles[it+1]=catFunc(statePathFiles[it]);
> } until (it==0);
> }
> 
> runLoop();
> ================
> 
> However, I get an error like this:
> 
> Running UC Eval
> Could not start execution.
> variable statePathFiles has multiple writers.
> 
> 
> Please suggest solutions for it .
> 
> Thank you
> Tibi
> 
> 
> 
> --
> Tiberiu (Tibi) Stef-Praun, PhD
> Computational Sciences Researcher
> Computation Institute
> 5640 S. Ellis Ave, #405
> University of Chicago
> http://www-unix.mcs.anl.gov/~tiberius/
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From benc at hawaga.org.uk  Wed Feb 17 10:58:42 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Wed, 17 Feb 2010 16:58:42 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
Message-ID: <Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>

> I have put together this really simple iteration example, which I expected
> to work (and it used to work in the past):

mmm.

yeah, the static analysis of when array elements are allowed to be 
assigned doesn't work well there.

If anyone is looking at fixing it, I think its something like the iterate 
statement being tagged as a writer of "all of statePathFiles", which then 
conflicts with the first line being a partial writer of statePathFiles. 

If anyone is *really* look at fixing it, get rid of foreach and iterate 
and put in proper dataflow based iteration (map, foldr, whatever) such 
that arrays are single-assignment.

>         statePathFiles[0]=initFunc("hello");
> 
>         iterate it{
>                 trace(@strcat("Iteration: ",it));
>                 statePathFiles[it+1]=catFunc(statePathFiles[it]);
>         } until (it==0);
> }

> Please suggest  solutions for it .

probably the two solutions above are not what you're looking for.

but:

move the statePathFiles[0] assignment into the iterate loop, by saying 
something like the below (but you'll have to adjust the array indices too, 
I think, because you need to start one earlier now). something like this:

iterate it { 
  if(it==0) {  statePathFiles[0]=initFunc("hello"); } else
   {statePathFiles[it+1]=catFunc(statePathFiles[it]);}
  } until(it==0);


-- 


From wilde at mcs.anl.gov  Wed Feb 17 11:05:21 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 17 Feb 2010 11:05:21 -0600 (CST)
Subject: [Swift-user] Re: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
Message-ID: <12659274.656851266426321107.JavaMail.root@zimbra>

An example of the "if 0" approach, Tibi:

   iterate r {
      if ( r==0 ) {
         pdata[0] = ItFixInit(p);
      }
      else {
         sim[r] = doRound(p, nsim, config, "runoops");
         (pdata[r], converged[r]) = analyze(sim[r], pdata[r-1], r, maxr);
      }
   } until ( r==maxr );

Ben, thanks for the suggested fix.

- Mike

----- "Ben Clifford" <benc at hawaga.org.uk> wrote:

> > I have put together this really simple iteration example, which I
> expected
> > to work (and it used to work in the past):
> 
> mmm.
> 
> yeah, the static analysis of when array elements are allowed to be 
> assigned doesn't work well there.
> 
> If anyone is looking at fixing it, I think its something like the
> iterate 
> statement being tagged as a writer of "all of statePathFiles", which
> then 
> conflicts with the first line being a partial writer of
> statePathFiles. 
> 
> If anyone is *really* look at fixing it, get rid of foreach and
> iterate 
> and put in proper dataflow based iteration (map, foldr, whatever) such
> 
> that arrays are single-assignment.
> 
> >         statePathFiles[0]=initFunc("hello");
> > 
> >         iterate it{
> >                 trace(@strcat("Iteration: ",it));
> >                 statePathFiles[it+1]=catFunc(statePathFiles[it]);
> >         } until (it==0);
> > }
> 
> > Please suggest  solutions for it .
> 
> probably the two solutions above are not what you're looking for.
> 
> but:
> 
> move the statePathFiles[0] assignment into the iterate loop, by saying
> 
> something like the below (but you'll have to adjust the array indices
> too, 
> I think, because you need to start one earlier now). something like
> this:
> 
> iterate it { 
>   if(it==0) {  statePathFiles[0]=initFunc("hello"); } else
>    {statePathFiles[it+1]=catFunc(statePathFiles[it]);}
>   } until(it==0);
> 
> 
> -- 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user


From wwj at ci.uchicago.edu  Wed Feb 17 11:56:17 2010
From: wwj at ci.uchicago.edu (Wenjun Wu)
Date: Wed, 17 Feb 2010 11:56:17 -0600
Subject: [Swift-devel] compilation problems with the latest swift package 
In-Reply-To: <Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
Message-ID: <4B7C2DC1.2010207@ci.uchicago.edu>

Hello,
  I tried to compiled the latest swift package from the swift svn and 
got some errors.
  The version info for the package is
 
svn info
Path: .
URL: https://svn.ci.uchicago.edu/svn/vdl2/trunk
Repository Root: https://svn.ci.uchicago.edu/svn/vdl2
Repository UUID: e2bb083e-7f23-0410-b3a8-8253ac9ef6d8
Revision: 3238
Node Kind: directory
Schedule: normal
Last Changed Author: wozniak
Last Changed Rev: 3237
Last Changed Date: 2010-02-12 15:32:49 -0500 (Fri, 12 Feb 2010)

 Could anyone give any suggestions for fixing the problem?
  Thanks,
Wenjun

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
  compile:
     [echo] [provider-coaster]: COMPILE
    [javac] /root/swift/cog/mbuild.xml:228: warning: 'includeantruntime' 
was not set, defaulting to build.sysclasspath=last; set to false for 
repeatable builds
    [javac] Compiling 6 source files to 
/root/swift/cog/modules/provider-coaster/build
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:88: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.InterruptedException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:132: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.InterruptedException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalCopyIOProvider.java:72: 
cannot find symbol
    [javac] symbol  : constructor IOException(java.net.URISyntaxException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:91: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.InterruptedException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:135: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.InterruptedException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:84: 
cannot find symbol
    [javac] symbol  : constructor IOException(java.lang.Exception)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException(e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:205: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.String,org.globus.cog.karajan.workflow.service.channels.ChannelException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException("Cannot establish 
channel to " + uri.getHost(), e);
    [javac]                       ^
    [javac] 
/root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:221: 
cannot find symbol
    [javac] symbol  : constructor 
IOException(java.lang.String,org.globus.cog.karajan.workflow.service.ProtocolException)
    [javac] location: class java.io.IOException
    [javac]                 throw new IOException("Error requesting file 
from " + channel, e);
    [javac]                       ^
    [javac] 8 errors

BUILD FAILED
/root/swift/cog/modules/swift/build.xml:73: The following error occurred 
while executing this line:
/root/swift/cog/mbuild.xml:444: The following error occurred while 
executing this line:
/root/swift/cog/mbuild.xml:79: The following error occurred while 
executing this line:
/root/swift/cog/mbuild.xml:52: The following error occurred while 
executing this line:
/root/swift/cog/modules/swift/dependencies.xml:13: The following error 
occurred while executing this line:
/root/swift/cog/mbuild.xml:163: The following error occurred while 
executing this line:
/root/swift/cog/mbuild.xml:168: The following error occurred while 
executing this line:
/root/swift/cog/modules/provider-coaster/build.xml:59: The following 
error occurred while executing this line:
/root/swift/cog/mbuild.xml:465: The following error occurred while 
executing this line:
/root/swift/cog/mbuild.xml:228: Compile failed; see the compiler error 
output for details.

Total time: 16 seconds



From hategan at mcs.anl.gov  Wed Feb 17 12:01:20 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 17 Feb 2010 12:01:20 -0600
Subject: [Swift-devel] compilation problems with the latest swift package
In-Reply-To: <4B7C2DC1.2010207@ci.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<4B7C2DC1.2010207@ci.uchicago.edu>
Message-ID: <1266429680.6048.1.camel@localhost>

Hi Wenjun,

We switched to java 1.5 with the trunk version of swift and cog.

The stable branch (details on the download page), which I would
recommend for you, is still made for 1.4.

Mihael

On Wed, 2010-02-17 at 11:56 -0600, Wenjun Wu wrote:
> Hello,
>   I tried to compiled the latest swift package from the swift svn and 
> got some errors.
>   The version info for the package is
>  
> svn info
> Path: .
> URL: https://svn.ci.uchicago.edu/svn/vdl2/trunk
> Repository Root: https://svn.ci.uchicago.edu/svn/vdl2
> Repository UUID: e2bb083e-7f23-0410-b3a8-8253ac9ef6d8
> Revision: 3238
> Node Kind: directory
> Schedule: normal
> Last Changed Author: wozniak
> Last Changed Rev: 3237
> Last Changed Date: 2010-02-12 15:32:49 -0500 (Fri, 12 Feb 2010)
> 
>  Could anyone give any suggestions for fixing the problem?
>   Thanks,
> Wenjun
> 
> /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>   compile:
>      [echo] [provider-coaster]: COMPILE
>     [javac] /root/swift/cog/mbuild.xml:228: warning: 'includeantruntime' 
> was not set, defaulting to build.sysclasspath=last; set to false for 
> repeatable builds
>     [javac] Compiling 6 source files to 
> /root/swift/cog/modules/provider-coaster/build
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:88: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.InterruptedException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:132: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.InterruptedException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalCopyIOProvider.java:72: 
> cannot find symbol
>     [javac] symbol  : constructor IOException(java.net.URISyntaxException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:91: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.InterruptedException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:135: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.InterruptedException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:84: 
> cannot find symbol
>     [javac] symbol  : constructor IOException(java.lang.Exception)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException(e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:205: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.channels.ChannelException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException("Cannot establish 
> channel to " + uri.getHost(), e);
>     [javac]                       ^
>     [javac] 
> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:221: 
> cannot find symbol
>     [javac] symbol  : constructor 
> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.ProtocolException)
>     [javac] location: class java.io.IOException
>     [javac]                 throw new IOException("Error requesting file 
> from " + channel, e);
>     [javac]                       ^
>     [javac] 8 errors
> 
> BUILD FAILED
> /root/swift/cog/modules/swift/build.xml:73: The following error occurred 
> while executing this line:
> /root/swift/cog/mbuild.xml:444: The following error occurred while 
> executing this line:
> /root/swift/cog/mbuild.xml:79: The following error occurred while 
> executing this line:
> /root/swift/cog/mbuild.xml:52: The following error occurred while 
> executing this line:
> /root/swift/cog/modules/swift/dependencies.xml:13: The following error 
> occurred while executing this line:
> /root/swift/cog/mbuild.xml:163: The following error occurred while 
> executing this line:
> /root/swift/cog/mbuild.xml:168: The following error occurred while 
> executing this line:
> /root/swift/cog/modules/provider-coaster/build.xml:59: The following 
> error occurred while executing this line:
> /root/swift/cog/mbuild.xml:465: The following error occurred while 
> executing this line:
> /root/swift/cog/mbuild.xml:228: Compile failed; see the compiler error 
> output for details.
> 
> Total time: 16 seconds
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel



From hategan at mcs.anl.gov  Wed Feb 17 13:14:45 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 17 Feb 2010 13:14:45 -0600
Subject: [Swift-devel] provider updates
Message-ID: <1266434085.9503.5.camel@localhost>

Given that the pdsh solution was flaky, I committed a patch (in the
branch) to launch multi jobs using ssh. It's a relatively accurate
replica of the code that Gram uses for the same purpose.

Also, there is now an SGE provider. I'm testing it now on ranger. If
anybody has access to other SGE sites, it may be useful to test there
since SGE seems a little less "uniform" than PBS to me.

You may need to specify a parallel environment (either by editing
etc/provider-sge.properties or by adding a "pe" attribute/profile to the
job/site)

Mihael



From hategan at mcs.anl.gov  Wed Feb 17 13:24:25 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 17 Feb 2010 13:24:25 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
Message-ID: <1266434665.9702.6.camel@localhost>

On Wed, 2010-02-17 at 16:58 +0000, Ben Clifford wrote:
> If anyone is *really* look at fixing it, get rid of foreach and iterate 
> and put in proper dataflow based iteration (map, foldr, whatever) such 
> that arrays are single-assignment.

Even in functional languages some of these problems are solved by
recursion rather than a single pre-defined function.

Which may or may not be fine as a usability issue, but it's almost
clearly not fine due to the lack of tail recursion optimization in
swift.



From hategan at mcs.anl.gov  Wed Feb 17 16:17:08 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 17 Feb 2010 16:17:08 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
Message-ID: <1266445028.16024.13.camel@localhost>

I committed a bunch of patches to trunk...

> 
> If anyone is looking at fixing it, I think its something like the iterate 
> statement being tagged as a writer of "all of statePathFiles", which then 
> conflicts with the first line being a partial writer of statePathFiles. 

That one. I'm not sure about the implications of it. Why was that flag
set to "not partial writer"? How did iterate ever work? What am I
missing?

> 
> If anyone is *really* look at fixing it, get rid of foreach and iterate 
> and put in proper dataflow based iteration (map, foldr, whatever) such 
> that arrays are single-assignment.

There's also a bunch of fixes that should enable the use of a normal
foreach for the same purpose.

Basically the code detects if iteration happens on a variable that is
updated in the body of the foreach and sets a special flag on the
foreach when that happens.

The foreach implementation then interprets (with some nasty details
about the concurrency involved which I hopefully got right) situations
with no running iterations as a sign that things are done. So the
following works as expected:

int a[];
a[0] = 50;

foreach v, k in a {
  trace(v);
  if (v > 0) {
    a[k + 1] = v - 1;
  }
}

Unfortunately static analysis fails to detect more complex patterns,
such as when there is an indirect dependence between the iteration
variable and the array that the body writes to. So the following
deadlocks:

int a[];
int b[];
a[0] = 50;

foreach v, k in a {
  trace(v);
  if (v > 0) {
    b[k + 1] = v - 1;
  }
}

foreach v, k in b {
  a[k] = b[k];
}

Ideas?



From benc at hawaga.org.uk  Thu Feb 18 02:46:17 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 18 Feb 2010 08:46:17 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266445028.16024.13.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>


> Why was that flag 
> set to "not partial writer"?

I think that was a bug rather than some deep reason.

> How did iterate ever work? What am I missing?

I think it may have only ever worked in the initialise-in-nested-if case
(or fairly trivial test cases which don't run into this problem)

> Ideas?

I'm split between two minds:

 a) more static and runtime analysis can lead to some (informally 
    provable?) correct behaviour here. It think it might be possible to at 
    least detect deadlocks caused by this and more elegantly error out, by 
    finding cycles in a graph of who is waiting for what, which is a 
    better user experience than a deadlock (as in, you can type the 
    deadlock message into google and get the beautiful documentation page 
    all about resolving such problems)

 b) that having multiple writer arrays and foreach/iterate control 
    structures in their present for are (informally provably?) inherently 
    deadlock prone and no matter what is done, it will always be possible 
    to contrive a deadlock; and so then to have a "correct" system, those 
    structures need to fundamentally change.

Even if b is the case, it might be that problems are sufficiently obscure 
that it doesn't matter from the "lets run a bazillion protein folds" 
perspective.

-- 


From benc at hawaga.org.uk  Thu Feb 18 08:11:37 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 18 Feb 2010 14:11:37 +0000 (GMT)
Subject: [Swift-devel] swiftscript language presentation
Message-ID: <Pine.LNX.4.64.1002181407450.22136@dildano.hawaga.org.uk>


I've been hanging out with the programming language seminar group at 
Utrecht University for fun(!) and so I gave a talk on the language side of 
Swift, quite a different focus from usual Swift presentations. Slides here 
if anyone is interested:  
http://www.hawaga.org.uk/ben/tech/swiftscript-uu.html

-- 
http://www.hawaga.org.uk/ben/


From wilde at mcs.anl.gov  Thu Feb 18 08:41:23 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 18 Feb 2010 08:41:23 -0600 (CST)
Subject: [Swift-devel] swiftscript language presentation
In-Reply-To: <Pine.LNX.4.64.1002181407450.22136@dildano.hawaga.org.uk>
Message-ID: <12328343.687281266504083116.JavaMail.root@zimbra>

Great talk & slides, Ben.

Could you post it somewhere prominent on the Swift main page?

Might be useful to separate out the parts that are purely tutorial and evolve it into a nice 5-minute intro to Swift.

- Mike

----- "Ben Clifford" <benc at hawaga.org.uk> wrote:

> I've been hanging out with the programming language seminar group at 
> Utrecht University for fun(!) and so I gave a talk on the language
> side of 
> Swift, quite a different focus from usual Swift presentations. Slides
> here 
> if anyone is interested:  
> http://www.hawaga.org.uk/ben/tech/swiftscript-uu.html
> 
> -- 
> http://www.hawaga.org.uk/ben/
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From benc at hawaga.org.uk  Thu Feb 18 08:45:22 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 18 Feb 2010 14:45:22 +0000 (GMT)
Subject: [Swift-devel] swiftscript language presentation
In-Reply-To: <12328343.687281266504083116.JavaMail.root@zimbra>
References: <12328343.687281266504083116.JavaMail.root@zimbra>
Message-ID: <Pine.LNX.4.64.1002181444450.15193@dildano.hawaga.org.uk>


> Could you post it somewhere prominent on the Swift main page?

well, I don't have access to that. But feel free to link to it yourself.

-- 



From hategan at mcs.anl.gov  Thu Feb 18 11:04:49 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 18 Feb 2010 11:04:49 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
Message-ID: <1266512689.6317.30.camel@localhost>

On Thu, 2010-02-18 at 08:46 +0000, Ben Clifford wrote:
> > Why was that flag 
> > set to "not partial writer"?
> 
> I think that was a bug rather than some deep reason.
> 
> > How did iterate ever work? What am I missing?
> 
> I think it may have only ever worked in the initialise-in-nested-if case
> (or fairly trivial test cases which don't run into this problem)
> 
> > Ideas?
> 
> I'm split between two minds:
> 
>  a) more static and runtime analysis can lead to some (informally 
>     provable?) correct behaviour here. It think it might be possible to at 
>     least detect deadlocks caused by this and more elegantly error out, by 
>     finding cycles in a graph of who is waiting for what, which is a 
>     better user experience than a deadlock (as in, you can type the 
>     deadlock message into google and get the beautiful documentation page 
>     all about resolving such problems)
> 
>  b) that having multiple writer arrays and foreach/iterate control 
>     structures in their present for are (informally provably?) inherently 
>     deadlock prone and no matter what is done, it will always be possible 
>     to contrive a deadlock; and so then to have a "correct" system, those 
>     structures need to fundamentally change.

I'm unsure. I have a feeling that static dealdock detection is
undecidable in swift. I need to think some more about that.

> 
> Even if b is the case, it might be that problems are sufficiently obscure 
> that it doesn't matter from the "lets run a bazillion protein folds" 
> perspective.
> 

Right. Which is what we've been doing so far.



From benc at hawaga.org.uk  Thu Feb 18 11:08:59 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 18 Feb 2010 17:08:59 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266512689.6317.30.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>


> I'm unsure. I have a feeling that static dealdock detection is
> undecidable in swift. I need to think some more about that.

I think doing it statically (i.e. at compile time) is equivalent to 
executing the code (including external apps) - I think there's probably a 
straightforward example where something depends on an integer read from an 
externally executed app which then influences an if statement that decides 
which of two different arrays is being written to; or something like that.

But I think it could be done at runtime (which is less pleasing, but still 
better than a runtime hang).

-- 


From hategan at mcs.anl.gov  Thu Feb 18 16:07:29 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 18 Feb 2010 16:07:29 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
Message-ID: <1266530849.15179.107.camel@localhost>

On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: 
> > I'm unsure. I have a feeling that static dealdock detection is
> > undecidable in swift. I need to think some more about that.
> 
> I think doing it statically (i.e. at compile time) is equivalent to 
> executing the code (including external apps) - I think there's probably a 
> straightforward example where something depends on an integer read from an 
> externally executed app which then influences an if statement that decides 
> which of two different arrays is being written to; or something like that.

Right. So termination is undecidable by virtue of it being a turing
complete language. What I'm unsure is whether deadlocks follow as a
consequence of that. They are a different beast.

We can probably start with simpler things. The following deadlocks with
the current swift:
a = cat(b);
b = cat(a);

So it seems that loops in the graph will create a deadlock and detecting
loops requires knowing the graph. We can probably detect the static
parts and we may be able to handle alternatives by issuing warnings of
the "branch 1 in that 'if' combined with branch 2 in that other 'if'
leads to a deadlock" sort.

Arrays may be a bit tougher.




From iraicu at cs.uchicago.edu  Thu Feb 18 16:17:22 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Thu, 18 Feb 2010 16:17:22 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266530849.15179.107.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	<1266445028.16024.13.camel@localhost>	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	<1266512689.6317.30.camel@localhost>	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
Message-ID: <4B7DBC72.6020403@cs.uchicago.edu>

But wouldn't the single assignment part prevent your example from 
dead-locking? In other words, the compiler should throw an error on line 
"b = cat(a)" as its trying to modify an existing value b.

Mihael Hategan wrote:
> On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: 
>   
>>> I'm unsure. I have a feeling that static dealdock detection is
>>> undecidable in swift. I need to think some more about that.
>>>       
>> I think doing it statically (i.e. at compile time) is equivalent to 
>> executing the code (including external apps) - I think there's probably a 
>> straightforward example where something depends on an integer read from an 
>> externally executed app which then influences an if statement that decides 
>> which of two different arrays is being written to; or something like that.
>>     
>
> Right. So termination is undecidable by virtue of it being a turing
> complete language. What I'm unsure is whether deadlocks follow as a
> consequence of that. They are a different beast.
>
> We can probably start with simpler things. The following deadlocks with
> the current swift:
> a = cat(b);
> b = cat(a);
>
> So it seems that loops in the graph will create a deadlock and detecting
> loops requires knowing the graph. We can probably detect the static
> parts and we may be able to handle alternatives by issuing warnings of
> the "branch 1 in that 'if' combined with branch 2 in that other 'if'
> leads to a deadlock" sort.
>
> Arrays may be a bit tougher.
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100218/e57dcadc/attachment.html>

From hategan at mcs.anl.gov  Thu Feb 18 16:37:43 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 18 Feb 2010 16:37:43 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7DBC72.6020403@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
Message-ID: <1266532663.17633.0.camel@localhost>

There is no existing value of b.

Here's the full example, which you can try out:

app (file y) cat(file x) {
	cat @x stdout=@y;
}

type file;

file a;
file b;

a = cat(b);
b = cat(a);

On Thu, 2010-02-18 at 16:17 -0600, Ioan Raicu wrote:
> But wouldn't the single assignment part prevent your example from
> dead-locking? In other words, the compiler should throw an error on
> line "b = cat(a)" as its trying to modify an existing value b.
> 
> Mihael Hategan wrote: 
> > On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: 
> >   
> > > > I'm unsure. I have a feeling that static dealdock detection is
> > > > undecidable in swift. I need to think some more about that.
> > > >       
> > > I think doing it statically (i.e. at compile time) is equivalent to 
> > > executing the code (including external apps) - I think there's probably a 
> > > straightforward example where something depends on an integer read from an 
> > > externally executed app which then influences an if statement that decides 
> > > which of two different arrays is being written to; or something like that.
> > >     
> > 
> > Right. So termination is undecidable by virtue of it being a turing
> > complete language. What I'm unsure is whether deadlocks follow as a
> > consequence of that. They are a different beast.
> > 
> > We can probably start with simpler things. The following deadlocks with
> > the current swift:
> > a = cat(b);
> > b = cat(a);
> > 
> > So it seems that loops in the graph will create a deadlock and detecting
> > loops requires knowing the graph. We can probably detect the static
> > parts and we may be able to handle alternatives by issuing warnings of
> > the "branch 1 in that 'if' combined with branch 2 in that other 'if'
> > leads to a deadlock" sort.
> > 
> > Arrays may be a bit tougher.
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > 
> >   
> 
> -- 
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384 
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu
> Web:   http://www.eecs.northwestern.edu/~iraicu/
>        https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> =================================================================
> 



From benc at hawaga.org.uk  Thu Feb 18 17:19:14 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Thu, 18 Feb 2010 23:19:14 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7DC3DF.4070107@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
Message-ID: <Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>



> But I don't get it, how can the variable b not have a value, if it is consumed
> by cat() in a prior statement?

Its not prior in execution order; its prior in source text, which does not 
imply execution order.

If I write (in maths, not in a program) this system of simultaneous 
equations:

x = y + 1
y = x + 1

then there is no solution giving values for x and y. Whats happening there 
is fairly close to what Mihael's example was (but saying +1 instead of 
cat, and using integers).

> I thought Swift followed a write-once read many
> paradigm, which was enforced by the single assignment paradigm.

yes (apart from arrays, which are monotonically assigned so have most of 
the same properties as single assignment variables)

> I guess there is a
> distinction between files and variables?

Variables hold "things". Those things can be in-memory values, or they can 
be files. But for this discussion the distinction doesn't matter - the 
file/in-memory value stuff is orthogonal to the data-directed execution.

-- 


From hategan at mcs.anl.gov  Thu Feb 18 20:13:29 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 18 Feb 2010 20:13:29 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
Message-ID: <1266545609.17911.3.camel@localhost>

On Thu, 2010-02-18 at 23:19 +0000, Ben Clifford wrote:
> 
> > But I don't get it, how can the variable b not have a value, if it is consumed
> > by cat() in a prior statement?
> 
> Its not prior in execution order; its prior in source text, which does not 
> imply execution order.
> 
> If I write (in maths, not in a program) this system of simultaneous 
> equations:
> 
> x = y + 1
> y = x + 1

A formulation which beautifully captures the nature of a Swift script.
"A set of simultaneous equations".



From benc at hawaga.org.uk  Fri Feb 19 04:53:31 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 19 Feb 2010 10:53:31 +0000 (GMT)
Subject: [Swift-devel] this paper
 (http://www.cs.uu.nl/research/techreps/UU-CS-2009-029.html) (fwd)
Message-ID: <Pine.LNX.4.64.1002191053000.15193@dildano.hawaga.org.uk>


In response to my swiftscript talk, one of the UU researchers sent me this 
article explaining why we should have implemented it in haskell ;)

-- 
http://www.hawaga.org.uk/ben/

---------- Forwarded message ----------
Date: Fri, 19 Feb 2010 11:35:50 +0100
From: S. Doaitse Swierstra <doaitse at swierstra.net>
To: benc at hawaga.org.uk
Cc: Wishnu&Ramosta Prasetya <wishnu at cs.uu.nl>
Subject: this paper (http://www.cs.uu.nl/research/techreps/UU-CS-2009-029.html)

I wrote on the occasion of 25 years of CS education to explain what we have been
doing and why. It seems as if it was written especially  for you ;-}

Best, and thanks once more for giving the talk,
Doaitse Swierstra

http://www.cs.uu.nl/research/techreps/UU-CS-2009-029.html


From benc at hawaga.org.uk  Fri Feb 19 05:13:55 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 19 Feb 2010 11:13:55 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266530849.15179.107.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>

> Right. So termination is undecidable by virtue of it being a turing
> complete language. What I'm unsure is whether deadlocks follow as a
> consequence of that. They are a different beast.

Turing complete languages don't deadlock in general.

(yes, concurrent programs do, but regard swift as a serial programming 
language, not as a concurrent one; and regard concurrent execution as 
'merely' an execution optimisation)

So I don't think deadlocks are a consequence of that.

But I do think that probably finding deadlocks given the language features 
of SwiftScript is undecideable (not as a consequence of turing 
completeness, but because of our choice of language features); and on a 
more practical note, requires knowledge of the output of app blocks.

-- 


From iraicu at cs.uchicago.edu  Fri Feb 19 10:48:56 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 10:48:56 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
Message-ID: <4B7EC0F8.7060005@cs.uchicago.edu>

But assuming there are dependencies, why wouldn't the order in source 
code also imply order in execution order?

Mathematically, the two different sequences evaluate to different values:

x = y + 1
y = x + 1

assuming y = 0, x = 1

you get:
x = 0 + 1 = 1
y = 1 + 1 = 2

final values for x = 1, y = 2

Now if the order was reversed:
y = x + 1
x = y + 1

assuming the same initial values y = 0, x = 1
y = 1 + 1 = 2
x = 2 + 1 = 3

Notice that the final value for x is different, depending on the order 
of the execution.

So, since the order matters (because there are dependencies between the 
2 statements, as if there were no dependencies the order would be 
irrelevant), then it makes sense that the execution order also follow 
the source code order, as the programmer likely wrote the code thinking 
that this order would be preserved.

So, I ask again, why would the execution order not follow the source 
code order, assuming there was some dependency among them?

Ioan

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




Ben Clifford wrote:
>   
>> But I don't get it, how can the variable b not have a value, if it is consumed
>> by cat() in a prior statement?
>>     
>
> Its not prior in execution order; its prior in source text, which does not 
> imply execution order.
>
> If I write (in maths, not in a program) this system of simultaneous 
> equations:
>
> x = y + 1
> y = x + 1
>
> then there is no solution giving values for x and y. Whats happening there 
> is fairly close to what Mihael's example was (but saying +1 instead of 
> cat, and using integers).
>
>   
>> I thought Swift followed a write-once read many
>> paradigm, which was enforced by the single assignment paradigm.
>>     
>
> yes (apart from arrays, which are monotonically assigned so have most of 
> the same properties as single assignment variables)
>
>   
>> I guess there is a
>> distinction between files and variables?
>>     
>
> Variables hold "things". Those things can be in-memory values, or they can 
> be files. But for this discussion the distinction doesn't matter - the 
> file/in-memory value stuff is orthogonal to the data-directed execution.
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/ad7950e2/attachment.html>

From benc at hawaga.org.uk  Fri Feb 19 10:51:39 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 19 Feb 2010 16:51:39 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EC0F8.7060005@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
Message-ID: <Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>


> Mathematically, the two different sequences evaluate to different values:
> 
> x = y + 1
> y = x + 1
> 
> assuming y = 0, x = 1

I mean in simultaneous equations (linear algebra) - in other words, "find 
(through whatever means you care to use) a value of x and y such that the 
above two equations are both satisfied" - there is no value of x and y 
that satisfies that.

-- 


From iraicu at cs.uchicago.edu  Fri Feb 19 11:01:55 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 11:01:55 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
Message-ID: <4B7EC403.9090307@cs.uchicago.edu>

Sent again from my UChicago adress:

Ioan Raicu wrote:
> But lets bring this back to a more real example. A user wanting to 
> express some computations that have some dependencies, would write out 
> their computations in some order, expecting their order to be 
> preserved because of the dependencies. If you only support single 
> assignment on variables (e.g. the data), then an example like the one 
> below could never deadlock because the single assignment would be 
> violated on the 2nd statement. Perhaps things are more complicated if 
> you support multiple assignments per variables, but that is not the 
> case for Swift, right?
>
> I am trying to understand if this deadlock is happening in Swift due 
> to some particular implementation detail in Swift (or underlying 
> pieces), or is it a fundamental flaw in the DAG based approach with 
> single assignment variables? Or is it due to something completely 
> different?
>
> Thanks,
> Ioan
> -- 
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384 
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu
> Web:   http://www.eecs.northwestern.edu/~iraicu/
>        https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> =================================================================
>
>   
>
>
> Ben Clifford wrote:
>>> Mathematically, the two different sequences evaluate to different values:
>>>
>>> x = y + 1
>>> y = x + 1
>>>
>>> assuming y = 0, x = 1
>>>     
>>
>> I mean in simultaneous equations (linear algebra) - in other words, "find 
>> (through whatever means you care to use) a value of x and y such that the 
>> above two equations are both satisfied" - there is no value of x and y 
>> that satisfies that.
>>
>>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/b043de14/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 11:27:18 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 11:27:18 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
Message-ID: <1266600438.22169.264.camel@localhost>

On Fri, 2010-02-19 at 11:13 +0000, Ben Clifford wrote:
> > Right. So termination is undecidable by virtue of it being a turing
> > complete language. What I'm unsure is whether deadlocks follow as a
> > consequence of that. They are a different beast.
> 
> Turing complete languages don't deadlock in general.
> 
> (yes, concurrent programs do, but regard swift as a serial programming 
> language, not as a concurrent one; and regard concurrent execution as 
> 'merely' an execution optimisation)

Swift semantics can be executed by a single thread without the
involvement of any concurrency as you point out. It would still lead to
non-termination due to loops in the dependency graph.

> 
> So I don't think deadlocks are a consequence of that.
> 

I wasn't asking whether deadlocks are a strict consequence of it being
turing complete. We can clearly have deadlocks with much simpler classes
of programs (a = f(b), b = f(a)).

What I was wondering is whether turing completeness implies
undecidability for deadlocks in the swift semantics.

> But I do think that probably finding deadlocks given the language features 
> of SwiftScript is undecideable (not as a consequence of turing 
> completeness, but because of our choice of language features); and on a 
> more practical note, requires knowledge of the output of app blocks.
> 

Swift is turing complete, so any algorithm that can be written in java
can be written in swift.

Say the deadlock finding function is D(program) returning true if
program deadlocks, false otherwise.

P(program) {
  if (D(program)) {
    return;
  }
  else {
    a = b + 1;
    b = a + 1;
  }
}

P(P):
D(P) = false -> P deadlocks
D(P) = true -> P returns

It's the textbook proof and does not involve apps or arrays.



From hategan at mcs.anl.gov  Fri Feb 19 11:34:39 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 11:34:39 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
Message-ID: <1266600879.22169.277.camel@localhost>

On Fri, 2010-02-19 at 10:59 -0600, Ioan Raicu wrote:
> But lets bring this back to a more real example. A user wanting to
> express some computations that have some dependencies, would write out
> their computations in some order, expecting their order to be
> preserved because of the dependencies. If you only support single
> assignment on variables (e.g. the data), then an example like the one
> below could never deadlock because the single assignment would be
> violated on the 2nd statement. Perhaps things are more complicated if
> you support multiple assignments per variables, but that is not the
> case for Swift, right? 

x = y + 1
y = x + 1

How many assignments are there? 2
How many variables are being assigned? 2
Do those assignments have the same lvalue (thing on the left)? no
---
Conclusion: each variable is assigned exactly once.

> 
> I am trying to understand if this deadlock is happening in Swift due
> to some particular implementation detail in Swift (or underlying
> pieces), or is it a fundamental flaw in the DAG based approach with
> single assignment variables? Or is it due to something completely
> different?

It seems to be a fundamental flaw with computers and algorithms.



From iraicu at cs.uchicago.edu  Fri Feb 19 11:36:00 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 11:36:00 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266600438.22169.264.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	<1266445028.16024.13.camel@localhost>	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	<1266512689.6317.30.camel@localhost>	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	<1266530849.15179.107.camel@localhost>	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
Message-ID: <4B7ECC00.1050102@cs.uchicago.edu>

But doesn't the single assignment criteria prevent loops from happening 
in the dependency graph?

Mihael Hategan wrote:
> On Fri, 2010-02-19 at 11:13 +0000, Ben Clifford wrote:
>   
>>> Right. So termination is undecidable by virtue of it being a turing
>>> complete language. What I'm unsure is whether deadlocks follow as a
>>> consequence of that. They are a different beast.
>>>       
>> Turing complete languages don't deadlock in general.
>>
>> (yes, concurrent programs do, but regard swift as a serial programming 
>> language, not as a concurrent one; and regard concurrent execution as 
>> 'merely' an execution optimisation)
>>     
>
> Swift semantics can be executed by a single thread without the
> involvement of any concurrency as you point out. It would still lead to
> non-termination due to loops in the dependency graph.
>
>   
>> So I don't think deadlocks are a consequence of that.
>>
>>     
>
> I wasn't asking whether deadlocks are a strict consequence of it being
> turing complete. We can clearly have deadlocks with much simpler classes
> of programs (a = f(b), b = f(a)).
>
> What I was wondering is whether turing completeness implies
> undecidability for deadlocks in the swift semantics.
>
>   
>> But I do think that probably finding deadlocks given the language features 
>> of SwiftScript is undecideable (not as a consequence of turing 
>> completeness, but because of our choice of language features); and on a 
>> more practical note, requires knowledge of the output of app blocks.
>>
>>     
>
> Swift is turing complete, so any algorithm that can be written in java
> can be written in swift.
>
> Say the deadlock finding function is D(program) returning true if
> program deadlocks, false otherwise.
>
> P(program) {
>   if (D(program)) {
>     return;
>   }
>   else {
>     a = b + 1;
>     b = a + 1;
>   }
> }
>
> P(P):
> D(P) = false -> P deadlocks
> D(P) = true -> P returns
>
> It's the textbook proof and does not involve apps or arrays.
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/b71b798e/attachment.html>

From foster at anl.gov  Fri Feb 19 11:39:25 2010
From: foster at anl.gov (Ian Foster)
Date: Fri, 19 Feb 2010 11:39:25 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7ECC00.1050102@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	<1266445028.16024.13.camel@localhost>	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	<1266512689.6317.30.camel@localhost>	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	<1266530849.15179.107.camel@localhost>	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
Message-ID: <19157226-753A-4784-9DDA-42DB3EE7B263@anl.gov>

No

On Feb 19, 2010, at 11:36 AM, Ioan Raicu wrote:

> But doesn't the single assignment criteria prevent loops from happening in the dependency graph?
> 
> Mihael Hategan wrote:
>> 
>> On Fri, 2010-02-19 at 11:13 +0000, Ben Clifford wrote:
>>   
>>>> Right. So termination is undecidable by virtue of it being a turing
>>>> complete language. What I'm unsure is whether deadlocks follow as a
>>>> consequence of that. They are a different beast.
>>>>       
>>> Turing complete languages don't deadlock in general.
>>> 
>>> (yes, concurrent programs do, but regard swift as a serial programming 
>>> language, not as a concurrent one; and regard concurrent execution as 
>>> 'merely' an execution optimisation)
>>>     
>> 
>> Swift semantics can be executed by a single thread without the
>> involvement of any concurrency as you point out. It would still lead to
>> non-termination due to loops in the dependency graph.
>> 
>>   
>>> So I don't think deadlocks are a consequence of that.
>>> 
>>>     
>> 
>> I wasn't asking whether deadlocks are a strict consequence of it being
>> turing complete. We can clearly have deadlocks with much simpler classes
>> of programs (a = f(b), b = f(a)).
>> 
>> What I was wondering is whether turing completeness implies
>> undecidability for deadlocks in the swift semantics.
>> 
>>   
>>> But I do think that probably finding deadlocks given the language features 
>>> of SwiftScript is undecideable (not as a consequence of turing 
>>> completeness, but because of our choice of language features); and on a 
>>> more practical note, requires knowledge of the output of app blocks.
>>> 
>>>     
>> 
>> Swift is turing complete, so any algorithm that can be written in java
>> can be written in swift.
>> 
>> Say the deadlock finding function is D(program) returning true if
>> program deadlocks, false otherwise.
>> 
>> P(program) {
>>   if (D(program)) {
>>     return;
>>   }
>>   else {
>>     a = b + 1;
>>     b = a + 1;
>>   }
>> }
>> 
>> P(P):
>> D(P) = false -> P deadlocks
>> D(P) = true -> P returns
>> 
>> It's the textbook proof and does not involve apps or arrays.
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>> 
>>   
> 
> -- 
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384 
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu
> Web:   http://www.eecs.northwestern.edu/~iraicu/
>        https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> =================================================================
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/d98e5b75/attachment.html>

From iraicu at cs.uchicago.edu  Fri Feb 19 11:41:37 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 11:41:37 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266600879.22169.277.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	
	<1266445028.16024.13.camel@localhost>	
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	
	<1266512689.6317.30.camel@localhost>	
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>	
	<1266532663.17633.0.camel@localhost>	
	<4B7DC3DF.4070107@eecs.northwestern.edu>	
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>	
	<4B7EC0F8.7060005@cs.uchicago.edu>	
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>	
	<4B7EC377.1030105@eecs.northwestern.edu>
	<1266600879.22169.277.camel@localhost>
Message-ID: <4B7ECD51.9040005@cs.uchicago.edu>


Mihael Hategan wrote:
> On Fri, 2010-02-19 at 10:59 -0600, Ioan Raicu wrote:
>   
>> But lets bring this back to a more real example. A user wanting to
>> express some computations that have some dependencies, would write out
>> their computations in some order, expecting their order to be
>> preserved because of the dependencies. If you only support single
>> assignment on variables (e.g. the data), then an example like the one
>> below could never deadlock because the single assignment would be
>> violated on the 2nd statement. Perhaps things are more complicated if
>> you support multiple assignments per variables, but that is not the
>> case for Swift, right? 
>>     
>
> x = y + 1
> y = x + 1
>
> How many assignments are there? 2
> How many variables are being assigned? 2
> Do those assignments have the same lvalue (thing on the left)? no
> ---
> Conclusion: each variable is assigned exactly once.
>   
However, y and x have to be defined somewhere, right? If they don't get 
their value assigned at definition time, then the statement x = y + 1 
should return with an error that y is not defined.
>   
>> I am trying to understand if this deadlock is happening in Swift due
>> to some particular implementation detail in Swift (or underlying
>> pieces), or is it a fundamental flaw in the DAG based approach with
>> single assignment variables? Or is it due to something completely
>> different?
>>     
>
> It seems to be a fundamental flaw with computers and algorithms.
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/b76d86f3/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 11:44:19 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 11:44:19 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7ECC00.1050102@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
Message-ID: <1266601459.22169.294.camel@localhost>

On Fri, 2010-02-19 at 11:36 -0600, Ioan Raicu wrote:
> But doesn't the single assignment criteria prevent loops from
> happening in the dependency graph?

No. What you are probably thinking is that if each node in a graph would
represent an assignment then the loop would only be executed once after
which, due to single assignment, there would be an error.

However the loop is not executed at all, because there is no way to
satisfy the dependencies of both nodes (x depends on y and y depends on
x).



From iraicu at cs.uchicago.edu  Fri Feb 19 11:50:37 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 11:50:37 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266601459.22169.294.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	
	<1266445028.16024.13.camel@localhost>	
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	
	<1266512689.6317.30.camel@localhost>	
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	
	<1266530849.15179.107.camel@localhost>	
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>	
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
Message-ID: <4B7ECF6D.5000009@cs.uchicago.edu>

I think we are on the same page, that if you cannot resolve the 
dependencies, things just fall apart. The part I don't get, is why would 
you get a deadlock in this situation (which is where this whole 
discussion started from), as opposed to an error detecting that the DAG 
cannot be constructed? It seems that at the time of DAG construction (I 
assume this is the SwiftScript compilation stage), any failure to add 
nodes to the DAG due to unresolved dependencies should return an error 
and halt, rather than execute until the deadlock happens.

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




Mihael Hategan wrote:
> On Fri, 2010-02-19 at 11:36 -0600, Ioan Raicu wrote:
>   
>> But doesn't the single assignment criteria prevent loops from
>> happening in the dependency graph?
>>     
>
> No. What you are probably thinking is that if each node in a graph would
> represent an assignment then the loop would only be executed once after
> which, due to single assignment, there would be an error.
>
> However the loop is not executed at all, because there is no way to
> satisfy the dependencies of both nodes (x depends on y and y depends on
> x).
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/d8b2b645/attachment.html>

From benc at hawaga.org.uk  Fri Feb 19 11:54:28 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 19 Feb 2010 17:54:28 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
Message-ID: <Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>


A point of information related to this discussion:

Underordered single assignment doesn't necessarily lead to deadlock in 
general. Using Haskell, and making this mutually recursive definition of x 
and y.

f l = 1 : l     -- if you like lisp,  1:l means (cons 1 l)

x = f y
y = f x

main = print x

x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so 
running the above code prints out a never-ending (until ctrl-c) list of 
1,1,1,1,...)

Also in Haskell, if you substitute in this,

f l = if length l < 200 then 1 : l else l

then Haskell detects a loop at runtime (not compile time) - that's pretty 
much the kind of "deadlock detection" that we're talking about here.

$ ./example2 
example2: <<loop>>

(or indeed anything involving the use of 'length l' I think)

It might be reasonable to not attempt to do something in this space if ghc 
cannot do it.

Now whats different between the above example and swiftscript? well, I 
used lists which is a data type that Swift doesn't have. Though maybe its 
possible to make something like this using arrays/structs somehow.

-- 
http://www.hawaga.org.uk/ben/



From benc at hawaga.org.uk  Fri Feb 19 11:57:22 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Fri, 19 Feb 2010 17:57:22 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7ECF6D.5000009@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
Message-ID: <Pine.LNX.4.64.1002191754460.15193@dildano.hawaga.org.uk>


> I think we are on the same page, that if you cannot resolve the 
> dependencies, things just fall apart.

yes.

> The part I don't get, is why would you get a deadlock
> in this situation (which is where this whole discussion started from), as
> opposed to an error detecting that the DAG cannot be constructed? It seems
> that at the time of DAG construction (I assume this is the SwiftScript
> compilation stage), any failure to add nodes to the DAG due to unresolved
> dependencies should return an error and halt, rather than execute until the
> deadlock happens.

So if you feed in an invalid program (i.e. not a program) then the 
behaviour is undefined - the actually-implemented behaviour and the 
in-your-head behaviour are both "reasonable" interpretations of what would 
happen in if you feed in an invalid program. There's nothing in the 
(informal) definition of SwiftScript compelling either behaviour.

-- 



From hategan at mcs.anl.gov  Fri Feb 19 12:04:30 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 12:04:30 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7ECF6D.5000009@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
Message-ID: <1266602670.27070.8.camel@localhost>

On Fri, 2010-02-19 at 11:50 -0600, Ioan Raicu wrote:
> I think we are on the same page, that if you cannot resolve the
> dependencies, things just fall apart. The part I don't get, is why
> would you get a deadlock in this situation (which is where this whole
> discussion started from), as opposed to an error detecting that the
> DAG cannot be constructed? It seems that at the time of DAG
> construction (I assume this is the SwiftScript compilation stage), any
> failure to add nodes to the DAG due to unresolved dependencies should
> return an error and halt, rather than execute until the deadlock
> happens.

a[0] = 0;
a[1] = a[2];
a[2] = a[complexFunctionThatReturns0or1()];

What should happen there?



From iraicu at cs.uchicago.edu  Fri Feb 19 12:28:09 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 12:28:09 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266602670.27070.8.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	<1266445028.16024.13.camel@localhost>	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	<1266512689.6317.30.camel@localhost>	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	<1266530849.15179.107.camel@localhost>	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost>
Message-ID: <4B7ED839.5030309@cs.uchicago.edu>

If you preserve the order of statements, statement 1 and 2 would get 
done first, in parallel if enough resources were available. However, 
statement 3 could not be evaluated until the complex function runs, at 
which time it can be added to the DAG and executed.

This implies that you don't need a complete DAG in order to execute the 
DAG. In other words, start building the DAG as much as you can with 
static information, but also start executing the DAG as soon as there is 
anything in the DAG. This would not work if you had to build the entire 
DAG apriori to starting to execute it, due to the dynamic nature of the 
data dependency.

So, your question about what should happen here, is that at run time you 
figure out the value of the complex function, and execute the statement 
accordingly. In the end, a[2] will have the same value as a[0] or a[1]. 
In theory, I don't see why this example would have to deadlock. Also, I 
don't think the order of the statements should be changed, for example 
moving statement statement 3 up and statement 2 down, preventing the 
a[2] overwrite. Assuming you cannot re-arrange the statements order when 
there are dependencies (and there are between statement 2 and 3), then 
this should work without any deadlocks.

Mihael Hategan wrote:
> On Fri, 2010-02-19 at 11:50 -0600, Ioan Raicu wrote:
>   
>> I think we are on the same page, that if you cannot resolve the
>> dependencies, things just fall apart. The part I don't get, is why
>> would you get a deadlock in this situation (which is where this whole
>> discussion started from), as opposed to an error detecting that the
>> DAG cannot be constructed? It seems that at the time of DAG
>> construction (I assume this is the SwiftScript compilation stage), any
>> failure to add nodes to the DAG due to unresolved dependencies should
>> return an error and halt, rather than execute until the deadlock
>> happens.
>>     
>
> a[0] = 0;
> a[1] = a[2];
> a[2] = a[complexFunctionThatReturns0or1()];
>
> What should happen there?
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/995900a6/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 12:58:56 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 12:58:56 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7ED839.5030309@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
Message-ID: <1266605936.29592.15.camel@localhost>

On Fri, 2010-02-19 at 12:28 -0600, Ioan Raicu wrote:
> If you preserve the order of statements, statement 1 and 2 would get
> done first, in parallel if enough resources were available.

How would you know the value of a[2] without the complex function in
statement 3 being evaluated?

>  However, statement 3 could not be evaluated until the complex
> function runs, at which time it can be added to the DAG and executed. 
> 
> This implies that you don't need a complete DAG in order to execute
> the DAG. In other words, start building the DAG as much as you can
> with static information, but also start executing the DAG as soon as
> there is anything in the DAG.

This is how swift works.

But then it's also the thing preventing you from checking the (not yet
fully built) dag at compile time.

>  This would not work if you had to build the entire DAG apriori to
> starting to execute it, due to the dynamic nature of the data
> dependency.
> 
> So, your question about what should happen here, is that at run time
> you figure out the value of the complex function, and execute the
> statement accordingly. In the end, a[2] will have the same value as
> a[0] or a[1]. In theory, I don't see why this example would have to
> deadlock.

Because if the complex function returns 1, then the program is the same
as (and forgive me for forgetting the dummy function, "+" in our case):

a[1] = a[2] + 1;
a[2] = a[1] + 1;

>  Also, I don't think the order of the statements should be changed,
> for example moving statement statement 3 up and statement 2 down,
> preventing the a[2] overwrite.

There is some fundamental misunderstanding here. There is no overwrite.
It's probably not useful to continue the discussion if you insist on its
existence. The confusion is probably up there with the "order of
execution of statements". There is no difference between any of the
permutations of the statements in that program. They all mean the same
thing. Assignment happens not when a statement is encountered in the
source code but when the value on the right is available. So in the case
of:
a = 0;
b = a + 1;
c = b + 1;

a = 0 happens at t0 (all constants are available at t0)
b = 1 happens at t0 + 1
c = 2 happens at t0 + 2

The exact same would happen if you wrote:
c = b + 1;
b = a + 1;
a = 0;

or
c = b + 1
a = 0
b = a + 1





From iraicu at cs.uchicago.edu  Fri Feb 19 14:09:25 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 14:09:25 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EEFD3.30902@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	
	<1266445028.16024.13.camel@localhost>	
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	
	<1266512689.6317.30.camel@localhost>	
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	
	<1266530849.15179.107.camel@localhost>	
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>	
	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>	
	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>	
	<1266602670.27070.8.camel@localhost>
	<4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
Message-ID: <4B7EEFF5.7070009@cs.uchicago.edu>

Oops, I sent it again from the wrong account.

Ioan Raicu wrote:
> Yes, I think my confusion is coming from the fact that the order of 
> statement in the source code is not preserved in the execution order 
> (assuming there are dependencies). You are saying that the order in 
> the source code doesn't matter in Swift. Perhaps this is what is 
> making the analysis harder, to detect these deadlocks? If Swift 
> insisted on an ordering in execution, relative to the source code 
> ordering, perhaps the analysis would be simplified?
>
> My curiosity is mostly driven by trying to understand what assumptions 
> in Swift is making this analysis hard to do, and how you would do 
> things differently to help this kind of analysis be easier to do? BTW, 
> I have a student in my class that is interested automated 
> parallelization of code, so this conversation is highly relevant for 
> his project (and my understanding).
>
> Thanks for all the explanations!!!
>
> Ioan
> -- 
> =================================================================
> Ioan Raicu, Ph.D.
> NSF/CRA Computing Innovation Fellow
> =================================================================
> Center for Ultra-scale Computing and Information Security (CUCIS)
> Department of Electrical Engineering and Computer Science
> Northwestern University
> 2145 Sheridan Rd, Tech M384 
> Evanston, IL 60208-3118
> =================================================================
> Cel:   1-847-722-0876
> Tel:   1-847-491-8163
> Email: iraicu at eecs.northwestern.edu
> Web:   http://www.eecs.northwestern.edu/~iraicu/
>        https://wiki.cucis.eecs.northwestern.edu/
> =================================================================
> =================================================================
>
>   
>
>
> Mihael Hategan wrote:
>> On Fri, 2010-02-19 at 12:28 -0600, Ioan Raicu wrote:
>>   
>>> If you preserve the order of statements, statement 1 and 2 would get
>>> done first, in parallel if enough resources were available.
>>>     
>>
>> How would you know the value of a[2] without the complex function in
>> statement 3 being evaluated?
>>
>>   
>>>  However, statement 3 could not be evaluated until the complex
>>> function runs, at which time it can be added to the DAG and executed. 
>>>
>>> This implies that you don't need a complete DAG in order to execute
>>> the DAG. In other words, start building the DAG as much as you can
>>> with static information, but also start executing the DAG as soon as
>>> there is anything in the DAG.
>>>     
>>
>> This is how swift works.
>>
>> But then it's also the thing preventing you from checking the (not yet
>> fully built) dag at compile time.
>>
>>   
>>>  This would not work if you had to build the entire DAG apriori to
>>> starting to execute it, due to the dynamic nature of the data
>>> dependency.
>>>
>>> So, your question about what should happen here, is that at run time
>>> you figure out the value of the complex function, and execute the
>>> statement accordingly. In the end, a[2] will have the same value as
>>> a[0] or a[1]. In theory, I don't see why this example would have to
>>> deadlock.
>>>     
>>
>> Because if the complex function returns 1, then the program is the same
>> as (and forgive me for forgetting the dummy function, "+" in our case):
>>
>> a[1] = a[2] + 1;
>> a[2] = a[1] + 1;
>>
>>   
>>>  Also, I don't think the order of the statements should be changed,
>>> for example moving statement statement 3 up and statement 2 down,
>>> preventing the a[2] overwrite.
>>>     
>>
>> There is some fundamental misunderstanding here. There is no overwrite.
>> It's probably not useful to continue the discussion if you insist on its
>> existence. The confusion is probably up there with the "order of
>> execution of statements". There is no difference between any of the
>> permutations of the statements in that program. They all mean the same
>> thing. Assignment happens not when a statement is encountered in the
>> source code but when the value on the right is available. So in the case
>> of:
>> a = 0;
>> b = a + 1;
>> c = b + 1;
>>
>> a = 0 happens at t0 (all constants are available at t0)
>> b = 1 happens at t0 + 1
>> c = 2 happens at t0 + 2
>>
>> The exact same would happen if you wrote:
>> c = b + 1;
>> b = a + 1;
>> a = 0;
>>
>> or
>> c = b + 1
>> a = 0
>> b = a + 1
>>
>>
>>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/68803d27/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 14:46:03 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 14:46:03 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EEFD3.30902@eecs.northwestern.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
Message-ID: <1266612363.31122.26.camel@localhost>

On Fri, 2010-02-19 at 14:08 -0600, Ioan Raicu wrote:
> Yes, I think my confusion is coming from the fact that the order of
> statement in the source code is not preserved in the execution order
> (assuming there are dependencies). You are saying that the order in
> the source code doesn't matter in Swift. Perhaps this is what is
> making the analysis harder, to detect these deadlocks?

Yes it does. It's also necessary because it's what achieves
parallelization.

>  If Swift insisted on an ordering in execution, relative to the source
> code ordering, perhaps the analysis would be simplified? 

Right. The analysis of deadlocks is very much simplified in languages
that don't allow concurrency as Ben pointed out (i.e. deadlocks don't
exist, being replaced by infinite loops of one form or another).

a = f(1);
b = f(2);

If b is evaluated after a has a value (which would be the execution
order you suggest) then f(1) and f(2) would not be evaluated in
parallel.

> 
> My curiosity is mostly driven by trying to understand what assumptions
> in Swift is making this analysis hard to do, and how you would do
> things differently to help this kind of analysis be easier to do?

Right. Of course. But it turns out that you can't have everything and to
me it looks like deadlocks in a parallel languages is a problem of the
same class as the halting problem in non-parallel languages.

This is pretty much what the earlier emails in the discussion were
about.

That said, there are trivial cases, such as the a > b, b > a case that
can be detected quickly.




From hategan at mcs.anl.gov  Fri Feb 19 15:01:16 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 15:01:16 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>
Message-ID: <1266613276.31696.1.camel@localhost>

On Fri, 2010-02-19 at 17:54 +0000, Ben Clifford wrote:
> A point of information related to this discussion:
> 
> Underordered single assignment doesn't necessarily lead to deadlock in 
> general. Using Haskell, and making this mutually recursive definition of x 
> and y.
> 
> f l = 1 : l     -- if you like lisp,  1:l means (cons 1 l)
> 
> x = f y
> y = f x
> 
> main = print x
> 
> x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so 
> running the above code prints out a never-ending (until ctrl-c) list of 
> 1,1,1,1,...)

What difference does that make? To the outside observer both programs
hang.



From iraicu at cs.uchicago.edu  Fri Feb 19 15:37:25 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 15:37:25 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266612363.31122.26.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	<1266445028.16024.13.camel@localhost>	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	<1266512689.6317.30.camel@localhost>	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	<1266530849.15179.107.camel@localhost>	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>	<1266602670.27070.8.camel@localhost>
	<4B7ED839.5030309@cs.uchicago.edu>	<1266605936.29592.15.camel@localhost>	<4B7EEFD3.30902@eecs.northwestern.edu>
	<1266612363.31122.26.camel@localhost>
Message-ID: <4B7F0495.9090207@cs.uchicago.edu>



Mihael Hategan wrote:
> On Fri, 2010-02-19 at 14:08 -0600, Ioan Raicu wrote:
>   
>> Yes, I think my confusion is coming from the fact that the order of
>> statement in the source code is not preserved in the execution order
>> (assuming there are dependencies). You are saying that the order in
>> the source code doesn't matter in Swift. Perhaps this is what is
>> making the analysis harder, to detect these deadlocks?
>>     
>
> Yes it does. It's also necessary because it's what achieves
> parallelization.
>   
Thinking about this problem briefly, I don't think its *necessary* for 
the parallelization. I can certainly think of how to execute a DAG in 
parallel (as long as there are no dependencies), and where-ever 
dependencies exist, things are done in the right order to maintain the 
dependencies. I believe this is what gets done in Swift right now. The 
difference is perhaps how you build the DAG. If you keep the order of 
statements from the source code, you just have to make sure that order 
is maintained when adding nodes (and edges) to the DAG.

For example:
b = f(a)
a = f(b)

You add the first node f_1 to the DAG, with a as input edge, and b as 
output edge. Then you process the second statement, adding another node 
f_2, adding an edge between f_1 and f_2 with dependency b, and create an 
output edge from f_2 with an edge a. The trick is to detect that edge a 
already existed, as an input to node f_1, which would cause a loop, 
which in turn should not be allowed (per the definition of a DAG), and 
an error should be raised.

I don't understand why you say its necessary to achieve parallelization. 
I am sure this discussion would be much simpler in person in front of 
whiteboard, or perhaps even in a skype call. Should we try that?
>   
>>  If Swift insisted on an ordering in execution, relative to the source
>> code ordering, perhaps the analysis would be simplified? 
>>     
>
> Right. The analysis of deadlocks is very much simplified in languages
> that don't allow concurrency as Ben pointed out (i.e. deadlocks don't
> exist, being replaced by infinite loops of one form or another).
>
> a = f(1);
> b = f(2);
>
> If b is evaluated after a has a value (which would be the execution
> order you suggest) then f(1) and f(2) would not be evaluated in
> parallel.
>   
but there are no dependencies between these 2 statements, so they can be 
executed in parallel, according to the way I built the DAG in my example 
above. You would have a DAG with 2 nodes, and no edges between the 
nodes. All nodes that have no unresolved dependencies should be able to 
execute at any time.
>   
>> My curiosity is mostly driven by trying to understand what assumptions
>> in Swift is making this analysis hard to do, and how you would do
>> things differently to help this kind of analysis be easier to do?
>>     
>
> Right. Of course. But it turns out that you can't have everything and to
> me it looks like deadlocks in a parallel languages is a problem of the
> same class as the halting problem in non-parallel languages.
>   
This is true in a general parallel language, but we are looking at a 
constraint case, in which we have single assignments, and loops aren't 
allowed as part of the DAG. So I am not convinced that deadlocks are 
innevitable, at a theoretical level, given all the assumptions we made 
about the parallel languages we care about (plus the ordering requirement).

Should we try a skype call? I can talk anytime up to 5PM today, or 
Monday any time.

Thanks,
Ioan
> This is pretty much what the earlier emails in the discussion were
> about.
>
> That said, there are trivial cases, such as the a > b, b > a case that
> can be detected quickly.
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/ffb2176f/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 16:19:28 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 16:19:28 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7F0495.9090207@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<1266612363.31122.26.camel@localhost>
	<4B7F0495.9090207@cs.uchicago.edu>
Message-ID: <1266617968.32175.21.camel@localhost>

On Fri, 2010-02-19 at 15:37 -0600, Ioan Raicu wrote:
> 
> 
> Mihael Hategan wrote: 
> > On Fri, 2010-02-19 at 14:08 -0600, Ioan Raicu wrote:
> >   
> > > Yes, I think my confusion is coming from the fact that the order of
> > > statement in the source code is not preserved in the execution order
> > > (assuming there are dependencies). You are saying that the order in
> > > the source code doesn't matter in Swift. Perhaps this is what is
> > > making the analysis harder, to detect these deadlocks?
> > >     
> > 
> > Yes it does. It's also necessary because it's what achieves
> > parallelization.
> >   
> Thinking about this problem briefly, I don't think its *necessary* for
> the parallelization.

Think a bit harder.

>  I can certainly think of how to execute a DAG in parallel (as long as
> there are no dependencies), and where-ever dependencies exist, things
> are done in the right order to maintain the dependencies. I believe
> this is what gets done in Swift right now. The difference is perhaps
> how you build the DAG. If you keep the order of statements from the
> source code, you just have to make sure that order is maintained when
> adding nodes (and edges) to the DAG. 
> 
> For example:
> b = f(a)
> a = f(b)
> 
> You add the first node f_1 to the DAG, with a as input edge, and b as
> output edge. Then you process the second statement, adding another
> node f_2, adding an edge between f_1 and f_2 with dependency b, and
> create an output edge from f_2 with an edge a. The trick is to detect
> that edge a already existed, as an input to node f_1, which would
> cause a loop, which in turn should not be allowed (per the definition
> of a DAG), and an error should be raised. 

Right. Except edges in dags are traditionally the dependencies. So if
you replace "edge" with "node" and the other way around in your above
statement, we may be closer to the generally accepted terminology. And
yes, this is precisely how swift works.

And yes, the trick is to detect loops in the dag. You can do it by
actually constructing a dag or using the nice method hinted about by Ben
(i.e. seeing this as a system of inequations):
b > a
a > b
(you can see ">" as "edge from left to right" if you want).

Now you realize I hope that you've evaluated nor b nor a when you
constructed the dag. Which is where you confusion seemed to be. And if
you re-order the two statements, you get the same dag. And I hope you
also realize that the order in which things are executed in the dag has
little to do with the order in which the edge and node declarations
appear.

> 
> I don't understand why you say its necessary to achieve
> parallelization. I am sure this discussion would be much simpler in
> person in front of whiteboard, or perhaps even in a skype call. Should
> we try that?

I've already spent too much time on this. Sorry.

But I think you are on the right track towards understanding how this
works. I think you should actually try to implement a simple system
(doesn't have to have the full swift semantics) based on what you said
above and see what problems you run into.

So far from the discussion it looks like you pretty much took the same
steps that we did when we first wrote swift, so I suspect that if you
try it out yourself you will learn better why things are the way they
are.




From iraicu at cs.uchicago.edu  Fri Feb 19 17:11:42 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Fri, 19 Feb 2010 17:11:42 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266617968.32175.21.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	
	<1266445028.16024.13.camel@localhost>	
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>	
	<1266512689.6317.30.camel@localhost>	
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>	
	<1266530849.15179.107.camel@localhost>	
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>	
	<1266600438.22169.264.camel@localhost>	<4B7ECC00.1050102@cs.uchicago.edu>	
	<1266601459.22169.294.camel@localhost>	<4B7ECF6D.5000009@cs.uchicago.edu>	
	<1266602670.27070.8.camel@localhost>
	<4B7ED839.5030309@cs.uchicago.edu>	
	<1266605936.29592.15.camel@localhost>	
	<4B7EEFD3.30902@eecs.northwestern.edu>	
	<1266612363.31122.26.camel@localhost>
	<4B7F0495.9090207@cs.uchicago.edu>
	<1266617968.32175.21.camel@localhost>
Message-ID: <4B7F1AAE.7040302@cs.uchicago.edu>

OK, I'll (I mean a student of mine) take the approach of implementing a 
simple system to create DAGs, and then execute them. I'll report back if 
I learn anything interesting.

Ioan

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================




Mihael Hategan wrote:
> On Fri, 2010-02-19 at 15:37 -0600, Ioan Raicu wrote:
>   
>> Mihael Hategan wrote: 
>>     
>>> On Fri, 2010-02-19 at 14:08 -0600, Ioan Raicu wrote:
>>>   
>>>       
>>>> Yes, I think my confusion is coming from the fact that the order of
>>>> statement in the source code is not preserved in the execution order
>>>> (assuming there are dependencies). You are saying that the order in
>>>> the source code doesn't matter in Swift. Perhaps this is what is
>>>> making the analysis harder, to detect these deadlocks?
>>>>     
>>>>         
>>> Yes it does. It's also necessary because it's what achieves
>>> parallelization.
>>>   
>>>       
>> Thinking about this problem briefly, I don't think its *necessary* for
>> the parallelization.
>>     
>
> Think a bit harder.
>
>   
>>  I can certainly think of how to execute a DAG in parallel (as long as
>> there are no dependencies), and where-ever dependencies exist, things
>> are done in the right order to maintain the dependencies. I believe
>> this is what gets done in Swift right now. The difference is perhaps
>> how you build the DAG. If you keep the order of statements from the
>> source code, you just have to make sure that order is maintained when
>> adding nodes (and edges) to the DAG. 
>>
>> For example:
>> b = f(a)
>> a = f(b)
>>
>> You add the first node f_1 to the DAG, with a as input edge, and b as
>> output edge. Then you process the second statement, adding another
>> node f_2, adding an edge between f_1 and f_2 with dependency b, and
>> create an output edge from f_2 with an edge a. The trick is to detect
>> that edge a already existed, as an input to node f_1, which would
>> cause a loop, which in turn should not be allowed (per the definition
>> of a DAG), and an error should be raised. 
>>     
>
> Right. Except edges in dags are traditionally the dependencies. So if
> you replace "edge" with "node" and the other way around in your above
> statement, we may be closer to the generally accepted terminology. And
> yes, this is precisely how swift works.
>
> And yes, the trick is to detect loops in the dag. You can do it by
> actually constructing a dag or using the nice method hinted about by Ben
> (i.e. seeing this as a system of inequations):
> b > a
> a > b
> (you can see ">" as "edge from left to right" if you want).
>
> Now you realize I hope that you've evaluated nor b nor a when you
> constructed the dag. Which is where you confusion seemed to be. And if
> you re-order the two statements, you get the same dag. And I hope you
> also realize that the order in which things are executed in the dag has
> little to do with the order in which the edge and node declarations
> appear.
>
>   
>> I don't understand why you say its necessary to achieve
>> parallelization. I am sure this discussion would be much simpler in
>> person in front of whiteboard, or perhaps even in a skype call. Should
>> we try that?
>>     
>
> I've already spent too much time on this. Sorry.
>
> But I think you are on the right track towards understanding how this
> works. I think you should actually try to implement a simple system
> (doesn't have to have the full swift semantics) based on what you said
> above and see what problems you run into.
>
> So far from the discussion it looks like you pretty much took the same
> steps that we did when we first wrote swift, so I suspect that if you
> try it out yourself you will learn better why things are the way they
> are.
>
>   

-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100219/75300a2f/attachment.html>

From hategan at mcs.anl.gov  Fri Feb 19 19:06:11 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 19 Feb 2010 19:06:11 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk>
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>
Message-ID: <1266627971.11280.1.camel@localhost>

On Fri, 2010-02-19 at 17:54 +0000, Ben Clifford wrote:
> A point of information related to this discussion:
> 
> Underordered single assignment doesn't necessarily lead to deadlock in 
> general. Using Haskell, and making this mutually recursive definition of x 
> and y.
> 
> f l = 1 : l     -- if you like lisp,  1:l means (cons 1 l)
> 
> x = f y
> y = f x
> 
> main = print x

Also, try this:

f l = length l : l
x = f y
y = f x
main = print x

;)



From benc at hawaga.org.uk  Sat Feb 20 04:17:39 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 10:17:39 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266613276.31696.1.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<4B7DBC72.6020403@cs.uchicago.edu>
	<1266532663.17633.0.camel@localhost>
	<4B7DC3DF.4070107@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002182310500.15193@dildano.hawaga.org.uk> 
	<4B7EC0F8.7060005@cs.uchicago.edu>
	<Pine.LNX.4.64.1002191650290.15193@dildano.hawaga.org.uk>
	<4B7EC377.1030105@eecs.northwestern.edu>
	<Pine.LNX.4.64.1002191744410.15193@dildano.hawaga.org.uk>
	<1266613276.31696.1.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002200951150.15193@dildano.hawaga.org.uk>


> > x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so 
> > running the above code prints out a never-ending (until ctrl-c) list of 
> > 1,1,1,1,...)
> 
> What difference does that make? To the outside observer both programs
> hang.

the print in the original hangs.

print (take 5 x)

doesn't hang, despite the fact that x is an infinite list. It prints 
[1,1,1,1,1]

My point basically is that there are some safe uses of cyclic definitions, 
in languages that although different bear some strong similarity to 
SwiftScript. It might be that this applies to SwiftScript too or it might 
be sufficiently different (eg in data structures and in evaluation 
semantics) that it doesn't apply.

-- 


From benc at hawaga.org.uk  Sat Feb 20 05:51:48 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 11:51:48 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <4B7EEFF5.7070009@cs.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
Message-ID: <Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>


> > how you would do things differently to help this kind of analysis be 
> > easier to do? BTW, I have a student in my class that is interested 
> > automated parallelization of code, so this conversation is highly 
> > relevant for his project (and my understanding).

I think that deadlocks will always be possible. Especially given 
yesterday's playing with Haskell, I'm fairly convinced that its "not easy" 
to find them.

But I think data and control structures can be written that will nudge 
programmers in the direction of not going that way; and that are easier to 
analyse.

I'm fairly sure operators that look like map/fold/scan in Haskell would 
make that easier. Certainly it would make things different.

What I mean there is, rather than having a foreach where the body of the 
loop can do anything it wants, make that body much more constrained in 
what it can do.

eg in haskell, I can say this:

$ ghci
Prelude> map  (\x -> 10 * x)   [1,2,3,4,5]
[10,20,30,40,50]

This is doing pretty much what a foreach is often used for. But the body 
(take a value and multiply by 10) is very constrained in what it can 
output. It cannot assign any values to variables - it can only output a 
value, which is then placed (by map) in the corresponding part of the 
output list.

So what would that do to SwiftScript? Well you could get rid of any 
partial assignment to arrays and say the equivalent of:

let a = map  (\x -> 10 * x)   [1,2,3,4,5]

Now a is single assignment, rather than more generally monotonically 
assigned. I think that is one part that is awkward in the present 
swiftscript.

But equally on the usability side, making people learn what a lambda 
expression is and how that fits into the above syntax might be awkward. 
Maybe its a price worth paying. A first approximation would probably be to 
make example syntax and see if its possible to explain to skenny or other 
similar character in a few paragraphs.

-- 


From benc at hawaga.org.uk  Sat Feb 20 06:18:56 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 12:18:56 +0000 (GMT)
Subject: [Swift-devel] [gt-dev] Google Summer of Code 2010 - Please
 contribute project ideas (fwd)
Message-ID: <Pine.LNX.4.64.1002201208240.22136@dildano.hawaga.org.uk>


This has been good before. I'm probably still interested in mentoring 
(seeing as it pays out beer money and a tshirt to the mentor) 
language-related projects:
 * actually implementing map/fold like operators 
would be one, if it was likely to be accepted as a contribution into the 
production codebase (which I'm not sure what the feel is for that); 

 * a prototype haskell (or other functional language) implementation in 
order to explore that space (with no intention that this would become 
production code - intention would be to give more practical experience for 
fuel in long-winded debates about whether SwiftScript would be better 
implemented inside some other langauge)

-- 
http://www.hawaga.org.uk/ben/

---------- Forwarded message ----------
Date: Fri, 19 Feb 2010 07:40:52 -0600
From: Borja Sotomayor <borja at borjanet.com>
To: gt-dev <gt-dev at lists.globus.org>
Subject: [gt-dev] Google Summer of Code 2010 - Please contribute project ideas

Hi everyone,

Globus has participated in Google Summer of Code
(http://code.google.com/soc/) for the last two years, giving us the
opportunity to work with talented students, expand our community, and
develop cool projects that would have otherwise languished on a wishlist
somewhere (see http://tinyurl.com/globus-gsoc09 for details on last
summer's projects, and http://tinyurl.com/globus-gsoc08 for the 2008
projects).

We will be applying again this year to be a mentoring organization in
Summer of Code, and a big part of the application is preparing a strong
list of student projects. I am writing to encourage members of the
Globus community to add cool and interesting project ideas to the
following wiki page:

	http://dev.globus.org/wiki/Google_Summer_of_Code_2010_Ideas

You will find more information on GSoC and detailed instructions on how
to add a GSoC project idea in the above URL. Even though new project
ideas are preferable, past GSoC mentors should feel free to re-offer any
idea that was not developed last year (see last year's list:
http://dev.globus.org/wiki/Google_Summer_of_Code_2009_Ideas)

The GSoC deadline for mentoring organizations is March 12th, but we
would like all project ideas to be added by March 10th so we can have
time to polish them up, ask committers for clarifications, etc.

If you have any questions, please don't hesitate to reply to this thread.

Cheers!
-- 
::::::::::::::::::::::::::::::::::::::::::::::::::::
Borja Sotomayor, University of Chicago
 Ph.D. Candidate, Department of Computer Science
 Ryerson 257-C, 1100 East 58th Street, Chicago, IL
 http://people.cs.uchicago.edu/~borja/
Haizea: http://haizea.cs.uchicago.edu/
????????????????????????????????????????????????????
    "Dis maschine vill run und run!"
          -- Kurt G?del (on the Turing Machine)
::::::::::::::::::::::::::::::::::::::::::::::::::::

From hategan at mcs.anl.gov  Sat Feb 20 12:25:22 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 20 Feb 2010 12:25:22 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
Message-ID: <1266690322.23029.1.camel@localhost>

On Sat, 2010-02-20 at 11:51 +0000, Ben Clifford wrote:

> So what would that do to SwiftScript? Well you could get rid of any 
> partial assignment to arrays and say the equivalent of:

I think that convergence loops (i.e. "do this repeatedly until some
non-trivial condition becomes true") would not be possible then unless
recursion was used.



From benc at hawaga.org.uk  Sat Feb 20 13:27:20 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 19:27:20 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266690322.23029.1.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>

> 
> > So what would that do to SwiftScript? Well you could get rid of any 
> > partial assignment to arrays and say the equivalent of:
> 
> I think that convergence loops (i.e. "do this repeatedly until some
> non-trivial condition becomes true") would not be possible then unless
> recursion was used.

I was imagining having a high level construct to replace "iterate" that 
looks quite like iterate but is more restricted in its use eg takes a 
some (laststate -> newstate) operation (the iterate body now) and takes 
some (newstate -> boolean) operation (the until clause now).

Having provided that construct as part of SwiftScript, you wouldn't need 
recursion.

Thats a way to do things that keeps it fairly like it is now. A further 
departure from where it is now is to say "well it should be a proper 
language, and recursion shouldn't hurt".

-- 


From hategan at mcs.anl.gov  Sat Feb 20 13:55:34 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 20 Feb 2010 13:55:34 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
Message-ID: <1266695734.23516.21.camel@localhost>

On Sat, 2010-02-20 at 19:27 +0000, Ben Clifford wrote:
> > 
> > > So what would that do to SwiftScript? Well you could get rid of any 
> > > partial assignment to arrays and say the equivalent of:
> > 
> > I think that convergence loops (i.e. "do this repeatedly until some
> > non-trivial condition becomes true") would not be possible then unless
> > recursion was used.
> 
> I was imagining having a high level construct to replace "iterate" that 
> looks quite like iterate but is more restricted in its use eg takes a 
> some (laststate -> newstate) operation (the iterate body now) and takes 
> some (newstate -> boolean) operation (the until clause now).

That would probably be better for most cases. Though I'm not sure how
exactly this would be expressed.

I think, in a sense, the choice of foreach (and array indices) instead
of map (and lists) was due to a desire to make this close to C because
of the belief that our target audience would then have an easier time
adapting to it.

Which is funny because the non-restrictive nature of C is, I think, what
makes it easy to write the wrong program which is what you are trying to
get rid of.

> 
> Having provided that construct as part of SwiftScript, you wouldn't need 
> recursion.

Right. Loops and recursion are equivalent.

> 
> Thats a way to do things that keeps it fairly like it is now. A further 
> departure from where it is now is to say "well it should be a proper 
> language, and recursion shouldn't hurt".
> 



From benc at hawaga.org.uk  Sat Feb 20 14:10:26 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 20:10:26 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266695734.23516.21.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>


> That would probably be better for most cases. Though I'm not sure how
> exactly this would be expressed.

If you're trying to keep syntax that looks vaguely like foreach and 
iterate now, I'm not sure what either would look like...

When I implemented iterate I toyed with the idea of having magic variables 
called old and new that you could access, like this:

iterate {
  new = f(old)
} until(checkwonderful(new))

-- 



From hategan at mcs.anl.gov  Sat Feb 20 14:10:34 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 20 Feb 2010 14:10:34 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266695734.23516.21.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
Message-ID: <1266696634.25100.7.camel@localhost>

On Sat, 2010-02-20 at 13:55 -0600, Mihael Hategan wrote:
> On Sat, 2010-02-20 at 19:27 +0000, Ben Clifford wrote:
> > > 
> > > > So what would that do to SwiftScript? Well you could get rid of any 
> > > > partial assignment to arrays and say the equivalent of:
> > > 
> > > I think that convergence loops (i.e. "do this repeatedly until some
> > > non-trivial condition becomes true") would not be possible then unless
> > > recursion was used.
> > 
> > I was imagining having a high level construct to replace "iterate" that 
> > looks quite like iterate but is more restricted in its use eg takes a 
> > some (laststate -> newstate) operation (the iterate body now) and takes 
> > some (newstate -> boolean) operation (the until clause now).
> 
> That would probably be better for most cases. Though I'm not sure how
> exactly this would be expressed.
> 
> I think, in a sense, the choice of foreach (and array indices) instead
> of map (and lists) was due to a desire to make this close to C because
> of the belief that our target audience would then have an easier time
> adapting to it.
> 
> Which is funny because the non-restrictive nature of C is, I think, what
> makes it easy to write the wrong program which is what you are trying to
> get rid of.

Which is probably why we have the ongoing debates about language vs.
library:

Achieving a goal in Swift requires knowledge of the inner workings of
Swift which are mostly hidden from the users (i.e. the incantation
needed to do something is not obvious and is usually discovered by
asking on the mailing list), whereas a library, while still requiring
knowledge of the inner workings, would at least give the users a better
path towards knowing those inner workings.

I think they are both bad, in that the knowledge of said inner workings
are complexities that we want to hide. Except the current language is
doing a poor job at not requiring that knowledge.



From hategan at mcs.anl.gov  Sat Feb 20 14:28:45 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 20 Feb 2010 14:28:45 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECC00.1050102@cs.uchicago.edu>
	<1266601459.22169.294.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
Message-ID: <1266697725.25100.25.camel@localhost>

On Sat, 2010-02-20 at 20:10 +0000, Ben Clifford wrote:
> > That would probably be better for most cases. Though I'm not sure how
> > exactly this would be expressed.
> 
> If you're trying to keep syntax that looks vaguely like foreach and 
> iterate now, I'm not sure what either would look like...
> 
> When I implemented iterate I toyed with the idea of having magic variables 
> called old and new that you could access, like this:
> 
> iterate {
>   new = f(old)
> } until(checkwonderful(new))
> 

Is that better than asserting in documentation that old === a[i] and new
=== a[i + 1]? Does it prevent a user somehow from writing other funny
stuff in the body of iterate (the syntax doesn't suggest so)?




From benc at hawaga.org.uk  Sat Feb 20 14:57:34 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 20:57:34 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266697725.25100.25.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002202056570.15193@dildano.hawaga.org.uk>


> Is that better than asserting in documentation that old === a[i] and new
> === a[i + 1]? 

not really - that's why I didn't go down that path.

-- 


From benc at hawaga.org.uk  Sat Feb 20 15:09:15 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 20 Feb 2010 21:09:15 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266697725.25100.25.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>

oh the other thing i thought about more recently as that the parameter to 
a map/iterate like thing doesn't have to be a literal lambda expression - 
it can be the name of a function.

then you have some thing that says:

typeA a[];
typeB b[];
a = map foo b

(typeA o) foo(typeB i) { ... }

Now that happily exists alongside a more complicated literal lambda 
expression syntax, and clearly separates the scope of the iteration body 
away from the containing scope.

Although that latter is a bad thing as you then don't get to read eg. 
parameter variables in the scope containing the map call. The haskell 
answer to that is partial application

a = map (foo p1) b

(typeA o) foo(typeP parameter1, typeB i) { ... }

But then partial application looks kinda strange if you aren't used to it, 
which is then a violation of the look-like-C principle.

So I'm not convinced by that approach either.

-- 
http://www.hawaga.org.uk/ben/



From hategan at mcs.anl.gov  Sat Feb 20 15:52:58 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 20 Feb 2010 15:52:58 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ECF6D.5000009@cs.uchicago.edu>
	<1266602670.27070.8.camel@localhost> <4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
	<Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
Message-ID: <1266702778.27073.4.camel@localhost>

On Sat, 2010-02-20 at 21:09 +0000, Ben Clifford wrote:
> oh the other thing i thought about more recently as that the parameter to 
> a map/iterate like thing doesn't have to be a literal lambda expression - 
> it can be the name of a function.
> 
> then you have some thing that says:
> 
> typeA a[];
> typeB b[];
> a = map foo b
> 
> (typeA o) foo(typeB i) { ... }
> 
> Now that happily exists alongside a more complicated literal lambda 
> expression syntax, and clearly separates the scope of the iteration body 
> away from the containing scope.
> 
> Although that latter is a bad thing as you then don't get to read eg. 
> parameter variables in the scope containing the map call. The haskell 
> answer to that is partial application
> 
> a = map (foo p1) b
> 
> (typeA o) foo(typeP parameter1, typeB i) { ... }
> 
> But then partial application looks kinda strange if you aren't used to it, 
> which is then a violation of the look-like-C principle.

And then ML would just have full static scoping, so whatever is visible
in the scope of foo's definition is visible in its invocation.





From benc at hawaga.org.uk  Sun Feb 21 02:37:58 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 21 Feb 2010 08:37:58 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266702778.27073.4.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
	<Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
	<1266702778.27073.4.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002210813420.15193@dildano.hawaga.org.uk>


> And then ML would just have full static scoping, so whatever is visible
> in the scope of foo's definition is visible in its invocation.

But that only works if your variables are visible. For static parameters, 
sure. But:

foreach p1 in [1,2,3] { foreach p2 in [10,20,30] { o[p1][p2] = p1 + p2 }}

o = map outerbody [1,2,3]

outerbody p = map innerbody [10,20,30]

innerbody q = q + ?

So you end up wanting to define innerbody inside the scope of outerbody,

outputbody p = let
  innerbody q = q + p
 in map innerbody [10,20,30]

which looks pretty un-c-like indeed. But I guess something like that is 
necessary if you want to access loop variables from nested loops (which I 
think is desirable).


From hategan at mcs.anl.gov  Sun Feb 21 13:28:59 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 21 Feb 2010 13:28:59 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002210813420.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7ED839.5030309@cs.uchicago.edu>
	<1266605936.29592.15.camel@localhost>
	<4B7EEFD3.30902@eecs.northwestern.edu>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
	<Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
	<1266702778.27073.4.camel@localhost>
	<Pine.LNX.4.64.1002210813420.15193@dildano.hawaga.org.uk>
Message-ID: <1266780539.28231.36.camel@localhost>

On Sun, 2010-02-21 at 08:37 +0000, Ben Clifford wrote:
> > And then ML would just have full static scoping, so whatever is visible
> > in the scope of foo's definition is visible in its invocation.
> 
> But that only works if your variables are visible. For static parameters, 
> sure. But:
> 
> foreach p1 in [1,2,3] { foreach p2 in [10,20,30] { o[p1][p2] = p1 + p2 }}
> 
> o = map outerbody [1,2,3]
> 
> outerbody p = map innerbody [10,20,30]

You're not using p there

> 
> innerbody q = q + ?

outerbody p = map innerbody zip [10, 20, 30] replicate 3 p
innerbody p q = p + q

Or you could deal with nested foreaches by merging the values into
tuples:

body p q = p + q
p = map body cross [1, 2, 3] [10, 20, 30]

where we'd define:
cross a b = map (\c -> zip b replicate length b c) a

Either way it kinda looks like gibberish if you're not used to it.

> 
> So you end up wanting to define innerbody inside the scope of outerbody,

Right, so we'd need proper closures.

> 
> outputbody p = let
>   innerbody q = q + p
>  in map innerbody [10,20,30]
> 
> which looks pretty un-c-like indeed.
>  But I guess something like that is 
> necessary if you want to access loop variables from nested loops (which I 
> think is desirable).

There's also the list comprehension way with C-like syntax, which
involves no array indices:

o = list(
  foreach p in [1, 2, 3] {
    list(
      foreach q in [10, 20, 30] {
        p + q;
      }
    )
  }
)



From benc at hawaga.org.uk  Sun Feb 21 13:39:32 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sun, 21 Feb 2010 19:39:32 +0000 (GMT)
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <1266780539.28231.36.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com> 
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk> 
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk> 
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
	<Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
	<1266702778.27073.4.camel@localhost>
	<Pine.LNX.4.64.1002210813420.15193@dildano.hawaga.org.uk>
	<1266780539.28231.36.camel@localhost>
Message-ID: <Pine.LNX.4.64.1002211935450.15193@dildano.hawaga.org.uk>


> There's also the list comprehension way with C-like syntax, which
> involves no array indices:

yeah, I had forgotten about that. It seems to keep the shape a bit better.

This is the same as your example (if I understand your example), in 
Haskell:

a = do
  p <- [1,2,3]
  return $ do
     q <- [10,20,30]
     return (p + q)

main = print a

-- 


From hategan at mcs.anl.gov  Sun Feb 21 14:33:16 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 21 Feb 2010 14:33:16 -0600
Subject: [Swift-devel] Problem with iterate
In-Reply-To: <Pine.LNX.4.64.1002211935450.15193@dildano.hawaga.org.uk>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<1266445028.16024.13.camel@localhost>
	<Pine.LNX.4.64.1002180832260.15193@dildano.hawaga.org.uk>
	<1266512689.6317.30.camel@localhost>
	<Pine.LNX.4.64.1002181706180.15193@dildano.hawaga.org.uk>
	<1266530849.15179.107.camel@localhost>
	<Pine.LNX.4.64.1002191109250.15193@dildano.hawaga.org.uk>
	<1266600438.22169.264.camel@localhost>
	<4B7EEFF5.7070009@cs.uchicago.edu>
	<Pine.LNX.4.64.1002201103160.22136@dildano.hawaga.org.uk>
	<1266690322.23029.1.camel@localhost>
	<Pine.LNX.4.64.1002201924080.15193@dildano.hawaga.org.uk>
	<1266695734.23516.21.camel@localhost>
	<Pine.LNX.4.64.1002202008270.15193@dildano.hawaga.org.uk>
	<1266697725.25100.25.camel@localhost>
	<Pine.LNX.4.64.1002202103190.15193@dildano.hawaga.org.uk>
	<1266702778.27073.4.camel@localhost>
	<Pine.LNX.4.64.1002210813420.15193@dildano.hawaga.org.uk>
	<1266780539.28231.36.camel@localhost>
	<Pine.LNX.4.64.1002211935450.15193@dildano.hawaga.org.uk>
Message-ID: <1266784396.29578.2.camel@localhost>

On Sun, 2010-02-21 at 19:39 +0000, Ben Clifford wrote:
> > There's also the list comprehension way with C-like syntax, which
> > involves no array indices:
> 
> yeah, I had forgotten about that. It seems to keep the shape a bit better.
> 
> This is the same as your example (if I understand your example), in 
> Haskell:
> 
> a = do
>   p <- [1,2,3]
>   return $ do
>      q <- [10,20,30]
>      return (p + q)
> 
> main = print a
> 

mike at blabla tmp$ ./a.out
[[11,21,31],[12,22,32],[13,23,33]]

It seems to do the right thing.



From wwj at ci.uchicago.edu  Mon Feb 22 11:50:02 2010
From: wwj at ci.uchicago.edu (Wenjun Wu)
Date: Mon, 22 Feb 2010 11:50:02 -0600
Subject: [Swift-devel] compilation problems with the latest swift package
In-Reply-To: <1266429680.6048.1.camel@localhost>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>	
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>	
	<4B7C2DC1.2010207@ci.uchicago.edu>
	<1266429680.6048.1.camel@localhost>
Message-ID: <4B82C3CA.8050400@ci.uchicago.edu>

Hi Mihael,
  Are you suggesting that I should use JDK1.5 to compile the swift package?
  Actually I'm using JDK1.5 and Ant 1.6.5. But I still got the same problem.
  Is there any other things I should try to make it work?
  Thanks,
Wenjun
> Hi Wenjun,
>
> We switched to java 1.5 with the trunk version of swift and cog.
>
> The stable branch (details on the download page), which I would
> recommend for you, is still made for 1.4.
>
> Mihael
>
> On Wed, 2010-02-17 at 11:56 -0600, Wenjun Wu wrote:
>   
>> Hello,
>>   I tried to compiled the latest swift package from the swift svn and 
>> got some errors.
>>   The version info for the package is
>>  
>> svn info
>> Path: .
>> URL: https://svn.ci.uchicago.edu/svn/vdl2/trunk
>> Repository Root: https://svn.ci.uchicago.edu/svn/vdl2
>> Repository UUID: e2bb083e-7f23-0410-b3a8-8253ac9ef6d8
>> Revision: 3238
>> Node Kind: directory
>> Schedule: normal
>> Last Changed Author: wozniak
>> Last Changed Rev: 3237
>> Last Changed Date: 2010-02-12 15:32:49 -0500 (Fri, 12 Feb 2010)
>>
>>  Could anyone give any suggestions for fixing the problem?
>>   Thanks,
>> Wenjun
>>
>> /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>   compile:
>>      [echo] [provider-coaster]: COMPILE
>>     [javac] /root/swift/cog/mbuild.xml:228: warning: 'includeantruntime' 
>> was not set, defaulting to build.sysclasspath=last; set to false for 
>> repeatable builds
>>     [javac] Compiling 6 source files to 
>> /root/swift/cog/modules/provider-coaster/build
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:88: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.InterruptedException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:132: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.InterruptedException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalCopyIOProvider.java:72: 
>> cannot find symbol
>>     [javac] symbol  : constructor IOException(java.net.URISyntaxException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:91: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.InterruptedException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:135: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.InterruptedException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:84: 
>> cannot find symbol
>>     [javac] symbol  : constructor IOException(java.lang.Exception)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException(e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:205: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.channels.ChannelException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException("Cannot establish 
>> channel to " + uri.getHost(), e);
>>     [javac]                       ^
>>     [javac] 
>> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:221: 
>> cannot find symbol
>>     [javac] symbol  : constructor 
>> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.ProtocolException)
>>     [javac] location: class java.io.IOException
>>     [javac]                 throw new IOException("Error requesting file 
>> from " + channel, e);
>>     [javac]                       ^
>>     [javac] 8 errors
>>
>> BUILD FAILED
>> /root/swift/cog/modules/swift/build.xml:73: The following error occurred 
>> while executing this line:
>> /root/swift/cog/mbuild.xml:444: The following error occurred while 
>> executing this line:
>> /root/swift/cog/mbuild.xml:79: The following error occurred while 
>> executing this line:
>> /root/swift/cog/mbuild.xml:52: The following error occurred while 
>> executing this line:
>> /root/swift/cog/modules/swift/dependencies.xml:13: The following error 
>> occurred while executing this line:
>> /root/swift/cog/mbuild.xml:163: The following error occurred while 
>> executing this line:
>> /root/swift/cog/mbuild.xml:168: The following error occurred while 
>> executing this line:
>> /root/swift/cog/modules/provider-coaster/build.xml:59: The following 
>> error occurred while executing this line:
>> /root/swift/cog/mbuild.xml:465: The following error occurred while 
>> executing this line:
>> /root/swift/cog/mbuild.xml:228: Compile failed; see the compiler error 
>> output for details.
>>
>> Total time: 16 seconds
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>     
>
>   



From hategan at mcs.anl.gov  Mon Feb 22 12:18:05 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 22 Feb 2010 12:18:05 -0600
Subject: [Swift-devel] compilation problems with the latest swift package
In-Reply-To: <4B82C3CA.8050400@ci.uchicago.edu>
References: <fec1351f1002170848sd0b45edt3c06e5986c76f69@mail.gmail.com>
	<Pine.LNX.4.64.1002171653110.22136@dildano.hawaga.org.uk>
	<4B7C2DC1.2010207@ci.uchicago.edu> <1266429680.6048.1.camel@localhost>
	<4B82C3CA.8050400@ci.uchicago.edu>
Message-ID: <1266862685.5873.2.camel@localhost>

My bad. IOException(Exception) was added in java 6.

Either use the stable branch or java 6.

On Mon, 2010-02-22 at 11:50 -0600, Wenjun Wu wrote:
> Hi Mihael,
>   Are you suggesting that I should use JDK1.5 to compile the swift package?
>   Actually I'm using JDK1.5 and Ant 1.6.5. But I still got the same problem.
>   Is there any other things I should try to make it work?
>   Thanks,
> Wenjun
> > Hi Wenjun,
> >
> > We switched to java 1.5 with the trunk version of swift and cog.
> >
> > The stable branch (details on the download page), which I would
> > recommend for you, is still made for 1.4.
> >
> > Mihael
> >
> > On Wed, 2010-02-17 at 11:56 -0600, Wenjun Wu wrote:
> >   
> >> Hello,
> >>   I tried to compiled the latest swift package from the swift svn and 
> >> got some errors.
> >>   The version info for the package is
> >>  
> >> svn info
> >> Path: .
> >> URL: https://svn.ci.uchicago.edu/svn/vdl2/trunk
> >> Repository Root: https://svn.ci.uchicago.edu/svn/vdl2
> >> Repository UUID: e2bb083e-7f23-0410-b3a8-8253ac9ef6d8
> >> Revision: 3238
> >> Node Kind: directory
> >> Schedule: normal
> >> Last Changed Author: wozniak
> >> Last Changed Rev: 3237
> >> Last Changed Date: 2010-02-12 15:32:49 -0500 (Fri, 12 Feb 2010)
> >>
> >>  Could anyone give any suggestions for fixing the problem?
> >>   Thanks,
> >> Wenjun
> >>
> >> /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
> >>   compile:
> >>      [echo] [provider-coaster]: COMPILE
> >>     [javac] /root/swift/cog/mbuild.xml:228: warning: 'includeantruntime' 
> >> was not set, defaulting to build.sysclasspath=last; set to false for 
> >> repeatable builds
> >>     [javac] Compiling 6 source files to 
> >> /root/swift/cog/modules/provider-coaster/build
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:88: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.InterruptedException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/CoGResourceIOProvider.java:132: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.InterruptedException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalCopyIOProvider.java:72: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor IOException(java.net.URISyntaxException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:91: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.InterruptedException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/LocalIOProvider.java:135: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.InterruptedException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:84: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor IOException(java.lang.Exception)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException(e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:205: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.channels.ChannelException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException("Cannot establish 
> >> channel to " + uri.getHost(), e);
> >>     [javac]                       ^
> >>     [javac] 
> >> /root/swift/cog/modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/handlers/providers/ProxyIOProvider.java:221: 
> >> cannot find symbol
> >>     [javac] symbol  : constructor 
> >> IOException(java.lang.String,org.globus.cog.karajan.workflow.service.ProtocolException)
> >>     [javac] location: class java.io.IOException
> >>     [javac]                 throw new IOException("Error requesting file 
> >> from " + channel, e);
> >>     [javac]                       ^
> >>     [javac] 8 errors
> >>
> >> BUILD FAILED
> >> /root/swift/cog/modules/swift/build.xml:73: The following error occurred 
> >> while executing this line:
> >> /root/swift/cog/mbuild.xml:444: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/mbuild.xml:79: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/mbuild.xml:52: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/modules/swift/dependencies.xml:13: The following error 
> >> occurred while executing this line:
> >> /root/swift/cog/mbuild.xml:163: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/mbuild.xml:168: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/modules/provider-coaster/build.xml:59: The following 
> >> error occurred while executing this line:
> >> /root/swift/cog/mbuild.xml:465: The following error occurred while 
> >> executing this line:
> >> /root/swift/cog/mbuild.xml:228: Compile failed; see the compiler error 
> >> output for details.
> >>
> >> Total time: 16 seconds
> >>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>     
> >
> >   
> 



From wilde at mcs.anl.gov  Thu Feb 25 10:00:41 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 25 Feb 2010 10:00:41 -0600 (CST)
Subject: [Swift-devel] PBS coasters miscalculate PBS options
In-Reply-To: <546154.891361267113197458.JavaMail.root@zimbra>
Message-ID: <10476330.891491267113641983.JavaMail.root@zimbra>

Mihael, running a 1000 job workflow with minimal specs in the sites.xml entry for coasters on PADS gave the error "(qsub reported an exit code of 188). 
qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes" (full trace below). The sites entry was:

  <pool handle="pbs">
    <profile namespace="globus" key="maxwalltime">00:00:10</profile>
    <profile namespace="globus" key="maxtime">1800</profile>
    <execution provider="coaster" url="none" jobManager="local:pbs"/>
    <profile namespace="globus" key="workersPerNode">1</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="karajan" key="jobThrottle">5.99</profile>
    <filesystem provider="local"/>
    <workdirectory>$(pwd)</workdirectory>
  </pool>

- Mike


Swift running in SwiftR.run.056 
Swift svn swift-r3202 cog-r2683

RunID: 20100225-0813-xn3bajnc
Progress:
Progress:  uninitialized:1
Progress:  Selecting site:399  Stage in:600  Submitting:1
Progress:  Selecting site:399  Stage in:529  Submitting:2  Submitted:70
Progress:  Selecting site:399  Stage in:413  Submitted:188
Progress:  Selecting site:399  Stage in:326  Submitted:275
Worker task failed: Error submitting block task
org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (qsub reported an exit code of 188). 
qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes

        at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)
        at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
        at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66)
Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (qsub reported an exit code of 188). 
qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes

        at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:86)
        at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)
        ... 3 more
Failed to shut down block
org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task
        at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:149)
        at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85)
        at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70)
        at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96)
        at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44)
        at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271)
        at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:151)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
Exception caught in block processor
java.util.ConcurrentModificationException
        at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
        at java.util.AbstractList$Itr.next(AbstractList.java:343)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:149)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
Exception caught in block processor
java.util.ConcurrentModificationException
        at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
        at java.util.AbstractList$Itr.next(AbstractList.java:343)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:149)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
        at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
Cleaning up...
Shutting down service at https://192.5.86.5:50002
Got channel MetaChannel: 1151109057 -> null
+Canceling job 4970.svc.pads.ci.uchicago.edu
Canceling job 4971.svc.pads.ci.uchicago.edu
Canceling job 4972.svc.pads.ci.uchicago.edu
Canceling job 4973.svc.pads.ci.uchicago.edu
Canceling job 4974.svc.pads.ci.uchicago.edu
 Done


From wilde at mcs.anl.gov  Thu Feb 25 10:11:14 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 25 Feb 2010 10:11:14 -0600 (CST)
Subject: [Swift-devel] PBS coasters miscalculate PBS options
In-Reply-To: <10476330.891491267113641983.JavaMail.root@zimbra>
Message-ID: <5559442.892631267114274479.JavaMail.root@zimbra>

I suspect that even though qstat on pads doesnt show node limits on the queue, it likely balks if you ask for more nodes than exist on the system. I'll try setting provider options that keep it below 48 nodes (or 384 nodes depending on how this is counted), or better yet much lower so that it doesnt create jobs that will never be runnable.

- Mike

login1$ qstat -q

server: svc.pads.ci.uchicago.edu

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
short              --      --    04:00:00   --    0   0 --   E R
extended           --      --       --      --    2   0 --   E R
fast               --      --    01:00:00   --    0   0 --   E R
long               --      --    24:00:00   --    0   0 --   E R
                                               ----- -----
                                                   2     0
login1$ 


----- wilde at mcs.anl.gov wrote:

> Mihael, running a 1000 job workflow with minimal specs in the
> sites.xml entry for coasters on PADS gave the error "(qsub reported an
> exit code of 188). 
> qsub: Job exceeds queue resource limits MSG=cannot locate feasible
> nodes" (full trace below). The sites entry was:
> 
>   <pool handle="pbs">
>     <profile namespace="globus" key="maxwalltime">00:00:10</profile>
>     <profile namespace="globus" key="maxtime">1800</profile>
>     <execution provider="coaster" url="none" jobManager="local:pbs"/>
>     <profile namespace="globus" key="workersPerNode">1</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="karajan" key="jobThrottle">5.99</profile>
>     <filesystem provider="local"/>
>     <workdirectory>$(pwd)</workdirectory>
>   </pool>
> 
> - Mike
> 
> 
> Swift running in SwiftR.run.056 
> Swift svn swift-r3202 cog-r2683
> 
> RunID: 20100225-0813-xn3bajnc
> Progress:
> Progress:  uninitialized:1
> Progress:  Selecting site:399  Stage in:600  Submitting:1
> Progress:  Selecting site:399  Stage in:529  Submitting:2 
> Submitted:70
> Progress:  Selecting site:399  Stage in:413  Submitted:188
> Progress:  Selecting site:399  Stage in:326  Submitted:275
> Worker task failed: Error submitting block task
> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
> Cannot submit job: Could not submit job (qsub reported an exit code of
> 188). 
> qsub: Job exceeds queue resource limits MSG=cannot locate feasible
> nodes
> 
>         at
> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)
>         at
> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46)
>         at
> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66)
> Caused by:
> org.globus.cog.abstraction.impl.scheduler.common.ProcessException:
> Could not submit job (qsub reported an exit code of 188). 
> qsub: Job exceeds queue resource limits MSG=cannot locate feasible
> nodes
> 
>         at
> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:86)
>         at
> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)
>         ... 3 more
> Failed to shut down block
> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
> Can only cancel an active task
>         at
> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:149)
>         at
> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85)
>         at
> org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70)
>         at
> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96)
>         at
> org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:151)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
> Exception caught in block processor
> java.util.ConcurrentModificationException
>         at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:149)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
> Exception caught in block processor
> java.util.ConcurrentModificationException
>         at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.cleanDoneBlocks(BlockQueueProcessor.java:149)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:436)
>         at
> org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:78)
> Cleaning up...
> Shutting down service at https://192.5.86.5:50002
> Got channel MetaChannel: 1151109057 -> null
> +Canceling job 4970.svc.pads.ci.uchicago.edu
> Canceling job 4971.svc.pads.ci.uchicago.edu
> Canceling job 4972.svc.pads.ci.uchicago.edu
> Canceling job 4973.svc.pads.ci.uchicago.edu
> Canceling job 4974.svc.pads.ci.uchicago.edu
>  Done
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Thu Feb 25 10:16:31 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 25 Feb 2010 10:16:31 -0600
Subject: [Swift-devel] PBS coasters miscalculate PBS options
In-Reply-To: <5559442.892631267114274479.JavaMail.root@zimbra>
References: <5559442.892631267114274479.JavaMail.root@zimbra>
Message-ID: <1267114591.21371.3.camel@localhost>

On Thu, 2010-02-25 at 10:11 -0600, Michael Wilde wrote:
> I suspect that even though qstat on pads doesnt show node limits on
> the queue, it likely balks if you ask for more nodes than exist on the
> system.

Yes, it will. It's a good idea to set "maxnodes" unless you have
sufficiently few jobs in the workflow.

>  I'll try setting provider options that keep it below 48 nodes (or 384
> nodes depending on how this is counted), or better yet much lower so
> that it doesnt create jobs that will never be runnable.

I'd set the maximum to the maximum allowed. The coaster manager will
submit jobs with fewer nodes.




From wilde at mcs.anl.gov  Thu Feb 25 13:26:17 2010
From: wilde at mcs.anl.gov (wilde at mcs.anl.gov)
Date: Thu, 25 Feb 2010 13:26:17 -0600 (CST)
Subject: [Swift-devel] Swift PBS coaster jobs seem to bypass automounter
In-Reply-To: <2675472.901491267125955526.JavaMail.root@zimbra>
Message-ID: <15747812.901541267125977064.JavaMail.root@zimbra>

Mihael, Neil Best is having problems accessing PADS datasets on the SEE project from Swift jobs run to the coaster pbs provider.

The same file access works fine from a simple qsub job or a qsub -I

The path that's unreachable/unmounted from a Swift PBS coaster job is:

  /gpfs/pads

In that dir, ls run under swift sees only:

drwxr-xr-x 5 root root 4096 Feb 25 12:35 scratch

...while it should see the files below. I'm debugging further to find out why.
Something different in the user startup processing under Swift than under plain qsub?

- Mike


login1$ ls -l 
total 10848
lrwxrwxrwx  1 root  root         21 Jun 17  2009 angle -> projects/CI-NCR000005/
-rw-r--r--  1 root  root    3491069 Jun  2  2009 bad-files
drwxr-xr-x  2 root  root      32768 Jan  9 08:53 cnfs/
lrwxrwxrwx  1 root  root         21 Jun 15  2009 esg -> projects/CI-ATM000007/
----------  1 root  root      20608 Feb 25 13:23 fileset.quota
lrwxrwxrwx  1 root  root         21 Jun 23  2009 flash -> projects/CI-AST000010/
drwxrwsr-x 32 root  fmri      32768 Jan  8 12:42 fmri/
lrwxrwxrwx  1 root  root         21 Jun 15  2009 gedi -> projects/CI-IBN000008/
lrwxrwxrwx  1 root  root         25 Jun 15  2009 gedi-dev -> projects/CI-IBN000008/dev
lrwxrwxrwx  1 root  root         21 Jun 19  2009 gridts -> projects/CI-CCR000016/
----------  1 root  root      51840 Feb 25 13:23 group.quota
lrwxrwxrwx  1 root  root         21 Jun 23  2009 i2u2 -> projects/CI-SEE000014/
lrwxrwxrwx  1 root  root         21 Jun 23  2009 ligandatlas -> projects/CI-MCB000012/
lrwxrwxrwx  1 root  root         21 Jun 17  2009 margoliash -> projects/CI-IBN000003/
lrwxrwxrwx  1 root  root         21 Jun 19  2009 media -> projects/CI-STA000015/
drwxrwsr-x 21 root  newslab   32768 Sep 12 11:30 newslab/
drwxrwsr-x  9 wilde oops      32768 Feb 24 13:59 oops/
lrwxrwxrwx  1 root  root         21 Jun 19  2009 osgedu -> projects/CI-SEE000004/
drwxr-xr-x 41 root  root      32768 Feb 16 21:25 projects/
drwxr-xr-x 31 root  root      32768 Feb 25 10:11 scratch/
lrwxrwxrwx  1 root  root         21 Jun 23  2009 swift -> projects/CI-CCR000013/
drwxr-xr-x  2 grog  root      32768 Jan  9 11:38 test/
----------  1 root  root      89728 Feb 25 13:23 user.quota
lrwxrwxrwx  1 root  root         21 Jun 19  2009 whitelab -> projects/CI-MCB000017/
login1$ pwd
/gpfs/pads
login1$ df .
Filesystem           1K-blocks      Used Available Use% Mounted on
nfs.ci.uchicago.edu:/exports/pads-gpfs
                     359446007808 136176995328 223269012480  38% /autonfs/gpfs-pads
login1$ 



From wilde at mcs.anl.gov  Thu Feb 25 13:30:41 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 25 Feb 2010 13:30:41 -0600 (CST)
Subject: [Swift-devel] Swift PBS coaster jobs seem to bypass automounter
In-Reply-To: <15747812.901541267125977064.JavaMail.root@zimbra>
Message-ID: <1040028.901891267126241672.JavaMail.root@zimbra>

As far as I can tell, the same problem occurs in the plain PBS provider.

I'll try to capture the pbs submit file and see if the error can be reproduced submitting the same file manually.

- Mike

----- wilde at mcs.anl.gov wrote:

> Mihael, Neil Best is having problems accessing PADS datasets on the
> SEE project from Swift jobs run to the coaster pbs provider.
> 
> The same file access works fine from a simple qsub job or a qsub -I
> 
> The path that's unreachable/unmounted from a Swift PBS coaster job
> is:
> 
>   /gpfs/pads
> 
> In that dir, ls run under swift sees only:
> 
> drwxr-xr-x 5 root root 4096 Feb 25 12:35 scratch
> 
> ...while it should see the files below. I'm debugging further to find
> out why.
> Something different in the user startup processing under Swift than
> under plain qsub?
> 
> - Mike
> 
> 
> login1$ ls -l 
> total 10848
> lrwxrwxrwx  1 root  root         21 Jun 17  2009 angle ->
> projects/CI-NCR000005/
> -rw-r--r--  1 root  root    3491069 Jun  2  2009 bad-files
> drwxr-xr-x  2 root  root      32768 Jan  9 08:53 cnfs/
> lrwxrwxrwx  1 root  root         21 Jun 15  2009 esg ->
> projects/CI-ATM000007/
> ----------  1 root  root      20608 Feb 25 13:23 fileset.quota
> lrwxrwxrwx  1 root  root         21 Jun 23  2009 flash ->
> projects/CI-AST000010/
> drwxrwsr-x 32 root  fmri      32768 Jan  8 12:42 fmri/
> lrwxrwxrwx  1 root  root         21 Jun 15  2009 gedi ->
> projects/CI-IBN000008/
> lrwxrwxrwx  1 root  root         25 Jun 15  2009 gedi-dev ->
> projects/CI-IBN000008/dev
> lrwxrwxrwx  1 root  root         21 Jun 19  2009 gridts ->
> projects/CI-CCR000016/
> ----------  1 root  root      51840 Feb 25 13:23 group.quota
> lrwxrwxrwx  1 root  root         21 Jun 23  2009 i2u2 ->
> projects/CI-SEE000014/
> lrwxrwxrwx  1 root  root         21 Jun 23  2009 ligandatlas ->
> projects/CI-MCB000012/
> lrwxrwxrwx  1 root  root         21 Jun 17  2009 margoliash ->
> projects/CI-IBN000003/
> lrwxrwxrwx  1 root  root         21 Jun 19  2009 media ->
> projects/CI-STA000015/
> drwxrwsr-x 21 root  newslab   32768 Sep 12 11:30 newslab/
> drwxrwsr-x  9 wilde oops      32768 Feb 24 13:59 oops/
> lrwxrwxrwx  1 root  root         21 Jun 19  2009 osgedu ->
> projects/CI-SEE000004/
> drwxr-xr-x 41 root  root      32768 Feb 16 21:25 projects/
> drwxr-xr-x 31 root  root      32768 Feb 25 10:11 scratch/
> lrwxrwxrwx  1 root  root         21 Jun 23  2009 swift ->
> projects/CI-CCR000013/
> drwxr-xr-x  2 grog  root      32768 Jan  9 11:38 test/
> ----------  1 root  root      89728 Feb 25 13:23 user.quota
> lrwxrwxrwx  1 root  root         21 Jun 19  2009 whitelab ->
> projects/CI-MCB000017/
> login1$ pwd
> /gpfs/pads
> login1$ df .
> Filesystem           1K-blocks      Used Available Use% Mounted on
> nfs.ci.uchicago.edu:/exports/pads-gpfs
>                      359446007808 136176995328 223269012480  38%
> /autonfs/gpfs-pads
> login1$ 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From wilde at mcs.anl.gov  Thu Feb 25 14:42:47 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 25 Feb 2010 14:42:47 -0600 (CST)
Subject: [Swift-devel] Swift PBS coaster jobs seem to bypass automounter
In-Reply-To: <1040028.901891267126241672.JavaMail.root@zimbra>
Message-ID: <11560525.906911267130567323.JavaMail.root@zimbra>

At the moment I'm fairly certain this is a PADS automount problem. Ive seen the same script work and fail; I finally saw the problem manually with qsub -I. It seems ot be occurring on sporadic PADS nodes. Ive reported it to Support.

- Mike

----- "Michael Wilde" <wilde at mcs.anl.gov> wrote:

> As far as I can tell, the same problem occurs in the plain PBS
> provider.
> 
> I'll try to capture the pbs submit file and see if the error can be
> reproduced submitting the same file manually.
> 
> - Mike
> 
> ----- wilde at mcs.anl.gov wrote:
> 
> > Mihael, Neil Best is having problems accessing PADS datasets on the
> > SEE project from Swift jobs run to the coaster pbs provider.
> > 
> > The same file access works fine from a simple qsub job or a qsub -I
> > 
> > The path that's unreachable/unmounted from a Swift PBS coaster job
> > is:
> > 
> >   /gpfs/pads
> > 
> > In that dir, ls run under swift sees only:
> > 
> > drwxr-xr-x 5 root root 4096 Feb 25 12:35 scratch
> > 
> > ...while it should see the files below. I'm debugging further to
> find
> > out why.
> > Something different in the user startup processing under Swift than
> > under plain qsub?
> > 
> > - Mike
> > 
> > 
> > login1$ ls -l 
> > total 10848
> > lrwxrwxrwx  1 root  root         21 Jun 17  2009 angle ->
> > projects/CI-NCR000005/
> > -rw-r--r--  1 root  root    3491069 Jun  2  2009 bad-files
> > drwxr-xr-x  2 root  root      32768 Jan  9 08:53 cnfs/
> > lrwxrwxrwx  1 root  root         21 Jun 15  2009 esg ->
> > projects/CI-ATM000007/
> > ----------  1 root  root      20608 Feb 25 13:23 fileset.quota
> > lrwxrwxrwx  1 root  root         21 Jun 23  2009 flash ->
> > projects/CI-AST000010/
> > drwxrwsr-x 32 root  fmri      32768 Jan  8 12:42 fmri/
> > lrwxrwxrwx  1 root  root         21 Jun 15  2009 gedi ->
> > projects/CI-IBN000008/
> > lrwxrwxrwx  1 root  root         25 Jun 15  2009 gedi-dev ->
> > projects/CI-IBN000008/dev
> > lrwxrwxrwx  1 root  root         21 Jun 19  2009 gridts ->
> > projects/CI-CCR000016/
> > ----------  1 root  root      51840 Feb 25 13:23 group.quota
> > lrwxrwxrwx  1 root  root         21 Jun 23  2009 i2u2 ->
> > projects/CI-SEE000014/
> > lrwxrwxrwx  1 root  root         21 Jun 23  2009 ligandatlas ->
> > projects/CI-MCB000012/
> > lrwxrwxrwx  1 root  root         21 Jun 17  2009 margoliash ->
> > projects/CI-IBN000003/
> > lrwxrwxrwx  1 root  root         21 Jun 19  2009 media ->
> > projects/CI-STA000015/
> > drwxrwsr-x 21 root  newslab   32768 Sep 12 11:30 newslab/
> > drwxrwsr-x  9 wilde oops      32768 Feb 24 13:59 oops/
> > lrwxrwxrwx  1 root  root         21 Jun 19  2009 osgedu ->
> > projects/CI-SEE000004/
> > drwxr-xr-x 41 root  root      32768 Feb 16 21:25 projects/
> > drwxr-xr-x 31 root  root      32768 Feb 25 10:11 scratch/
> > lrwxrwxrwx  1 root  root         21 Jun 23  2009 swift ->
> > projects/CI-CCR000013/
> > drwxr-xr-x  2 grog  root      32768 Jan  9 11:38 test/
> > ----------  1 root  root      89728 Feb 25 13:23 user.quota
> > lrwxrwxrwx  1 root  root         21 Jun 19  2009 whitelab ->
> > projects/CI-MCB000017/
> > login1$ pwd
> > /gpfs/pads
> > login1$ df .
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > nfs.ci.uchicago.edu:/exports/pads-gpfs
> >                      359446007808 136176995328 223269012480  38%
> > /autonfs/gpfs-pads
> > login1$ 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Thu Feb 25 14:47:33 2010
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 25 Feb 2010 14:47:33 -0600
Subject: [Swift-devel] Swift PBS coaster jobs seem to bypass automounter
In-Reply-To: <11560525.906911267130567323.JavaMail.root@zimbra>
References: <11560525.906911267130567323.JavaMail.root@zimbra>
Message-ID: <1267130853.28684.0.camel@localhost>

On Thu, 2010-02-25 at 14:42 -0600, Michael Wilde wrote:
> At the moment I'm fairly certain this is a PADS automount problem. Ive
> seen the same script work and fail; I finally saw the problem manually
> with qsub -I. It seems ot be occurring on sporadic PADS nodes. Ive
> reported it to Support.

It's probably useless to say this now, but it was my initial suspicion.




From wilde at mcs.anl.gov  Thu Feb 25 14:50:38 2010
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 25 Feb 2010 14:50:38 -0600 (CST)
Subject: [Swift-devel] Swift PBS coaster jobs seem to bypass automounter
In-Reply-To: <1267130853.28684.0.camel@localhost>
Message-ID: <3258486.907501267131038460.JavaMail.root@zimbra>

The confusing thing was that qsub -I always seemed to get good nodes, and swift always seemed to get bad nodes. Finally more jobs got into the queue and seemed to change the  node allocation pattern.


----- "Mihael Hategan" <hategan at mcs.anl.gov> wrote:

> On Thu, 2010-02-25 at 14:42 -0600, Michael Wilde wrote:
> > At the moment I'm fairly certain this is a PADS automount problem.
> Ive
> > seen the same script work and fail; I finally saw the problem
> manually
> > with qsub -I. It seems ot be occurring on sporadic PADS nodes. Ive
> > reported it to Support.
> 
> It's probably useless to say this now, but it was my initial
> suspicion.


From iraicu at cs.uchicago.edu  Thu Feb 25 17:06:33 2010
From: iraicu at cs.uchicago.edu (Ioan Raicu)
Date: Thu, 25 Feb 2010 17:06:33 -0600
Subject: [Swift-devel] ACM ScienceCloud2010 deadline extension to 03/15/10
Message-ID: <4B870279.1050406@cs.uchicago.edu>

Call for Papers

---------------------------------------------------------------------------------------
1st ACM Workshop on Scientific Cloud Computing (ScienceCloud) 2010
http://dsl.cs.uchicago.edu/ScienceCloud2010/  
---------------------------------------------------------------------------------------
June 21st, 2010
Chicago, Illinois, USA

Co-located with with ACM High Performance Distributed Computing Conference (HPDC) 2010 

Topics of Interest
---------------------------------------------------------------------------------------
We invite the submission of original work that is related to the topics below. The papers 
can be either short (5 pages) position papers, or long (10 pages) research papers. Topics 
of interest include (in the context of Cloud Computing):
*       scientific computing applications
*       performance evaluation
*       programming models and tools
*       storage cloud architectures and implementations
*       compute resource management 
*       high-performance computing
*       models, frameworks and systems for cloud security

Important Dates
---------------------------------------------------------------------------------------
*       Abstract Due:                   March 15th, 2010
*       Papers Due:                     March 15th, 2010
*       Notification of Acceptance:     April 15th, 2010
*       Camera Ready Papers Due:        May 1st, 2010
*       Workshop Date:                  June 21st, 2010

Committee Members
---------------------------------------------------------------------------------------
Workshop Chairs
*       Pete Beckman, University of Chicago & Argonne National Laboratory
*       Ian Foster, University of Chicago & Argonne National Laboratory
*       Ioan Raicu, Northwestern University

Steering Committee
*       Jeff Broughton, Lawrence Berkeley National Lab., USA 
*       Alok Choudhary, Northwestern University, USA
*       Dennis Gannon, Microsoft Research, USA
*       Robert Grossman, University of Illinois at Chicago, USA
*       Kate Keahey, Nimbus, University of Chicago, Argonne National Laboratory, USA 
*       Ed Lazowska, University of Washington, USA
*       Ignacio Llorente, Open Nebula, Universidad Complutense de Madrid, Spain
*       David E. Martin, Argonne National Laboratory, Northwestern University, USA
*       Gabriel Mateescu, Linkoping University, Sweden 
*       David O'Hallaron, Carnegie Mellon University, Intel Labs, USA 
*       Rich Wolski, Eucalyptus, University of California, Santa Barbara, USA
*       Kathy Yelick, University of California at Berkeley, Lawrence Berkeley National Lab., USA



-- 
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384 
Evanston, IL 60208-3118
=================================================================
Cel:   1-847-722-0876
Tel:   1-847-491-8163
Email: iraicu at eecs.northwestern.edu
Web:   http://www.eecs.northwestern.edu/~iraicu/
       https://wiki.cucis.eecs.northwestern.edu/
=================================================================
=================================================================


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100225/8cc3ced0/attachment.html>

From benc at hawaga.org.uk  Sat Feb 27 06:54:06 2010
From: benc at hawaga.org.uk (Ben Clifford)
Date: Sat, 27 Feb 2010 12:54:06 +0000 (GMT)
Subject: [Swift-devel] [provenance-challenge] WANDS'10 workshop -- deadline
 extension (fwd)
Message-ID: <Pine.LNX.4.64.1002271253390.22136@dildano.hawaga.org.uk>


this was on the provenance challenge list

-- 
http://www.hawaga.org.uk/ben/

---------- Forwarded message ----------
Date: Sat, 27 Feb 2010 12:35:56 +0000
From: Paolo Missier <pmissier at cs.man.ac.uk>
Reply-To: provenance-challenge at ipaw.info
To: mygrid at listserv.manchester.ac.uk,
    myexperiment-discuss <myexperiment-discuss at nongnu.org>,
    provenance-challenge at ipaw.info, public-xg-prov at w3.org
Subject: [provenance-challenge]  WANDS'10 workshop -- deadline extension

#include <standard apologies for cross-posting>  :-)

for those of you who wisely planned to submit to the WANDS'10 workshop: please
note that the new deadline is *March 12th*.

Complete updated CFP attached. Please help creating good karma by spreading the
word

Cheers, -Paolo
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: WANDS-CFP-plaintext.txt
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100227/2e073bd1/attachment.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pmissier.vcf
Type: text/x-vcard
Size: 386 bytes
Desc: 
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20100227/2e073bd1/attachment.vcf>