From jamalphd at gmail.com Mon Feb 1 20:14:36 2010 From: jamalphd at gmail.com (J A) Date: Mon, 1 Feb 2010 21:14:36 -0500 Subject: [Swift-user] passing a file as an argument In-Reply-To: <27622487.155821264962002707.JavaMail.root@zimbra> References: <27622487.155821264962002707.JavaMail.root@zimbra> Message-ID: Hi: I tried that but got an error ... *the command is:* $ swift CreateFiles.swift -fname=list.txt *My code looks like this:* type file{}; type messagefile {}; //file list<"list.txt">; // if I uncomment this line and comment the next line, my code will work fine but I want the file name // to be dynamic file list<@arg("fname")>; .. ... ... *The error I got was:* $ swift CreateFiles.swift -fname=list.txt Could not compile SwiftScript source: line 7:11: unexpected token: @ Thanks for your help. Jamal On 1/31/10, Michael Wilde wrote: > > Jamal, for this, you pass the file name on the command line as a script > argument after all the swift command arguments, and pick it up inside your > swift script with the @arg() function, which is like argv in C (except > indexed by name rather than position). Its described in the User Guide. > > - Mike > > > ----- "J A" wrote: > > > Hi: > > > > how can I pass a file when executing swift, so i want to do something > > like this: > > > > $ swift file.txt > > > > > > how do I catch the file inside the code? > > > > Thanks, > > Jamal > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fedorov at bwh.harvard.edu Mon Feb 1 22:22:05 2010 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 1 Feb 2010 23:22:05 -0500 Subject: [Swift-user] Looking for the cause of failure In-Reply-To: <1264953402.11359.1.camel@localhost> References: <12680489.151301264897544218.JavaMail.root@zimbra> <2795624.151361264897648127.JavaMail.root@zimbra> <82f536811001302007v6940936fn37b9cd79bbab4736@mail.gmail.com> <1264911244.7775.3.camel@localhost> <82f536811001302028p6c58a241l87c4cd9af220c6f8@mail.gmail.com> <1264913153.8312.15.camel@localhost> <82f536811001310749o37a37509i6606f0f6bfcf3be3@mail.gmail.com> <1264953402.11359.1.camel@localhost> Message-ID: <82f536811002012022g1725800fnd488f88cedb48455@mail.gmail.com> Hi Mihael, pdsh does not seem to be available via softenv. I compiled it from source. My fresh errors are the following: "Worker task failed: 0201-560951-000000 Block task ended prematurely" and "pdsh at abe1177: abe1172: rcmd: socket: Permission denied" Do I need to perform some special setup of pdsh? Meanwhile, my exercises with stick lead me to conclusion that I should rather use "gt2:gt2:pbs" than "local:pbs" jobmanager in coaster configuration.... -- Andriy Fedorov, Ph.D. Research Fellow Brigham and Women's Hospital Harvard Medical School 75 Francis Street Boston, MA 02115 USA fedorov at bwh.harvard.edu On Sun, Jan 31, 2010 at 10:56, Mihael Hategan wrote: > On Sun, 2010-01-31 at 10:49 -0500, Andriy Fedorov wrote: >> On Sat, Jan 30, 2010 at 23:45, Mihael Hategan wrote: >> >> With the previous setup, it made more sense, because the number of >> >> active jobs was *. >> > >> > Define "previous setup". >> >> "previous setup" is the site configuration I included in the email >> that started this thread. >> >> I just tried this "previous setup", increasing number of workers per >> node to 8, and everything worked very well (job status plot attached). >> >> > If it's about one coaster job per node, yes. >> > Unfortunately that's also something that prevents scalability with gram2 >> > or clusters that have limits on the number of jobs in the queue (like >> > the BG/P). >> > >> > You can force that behavior though with maxnodes=1. >> > >> >> >> >> Am I missing something simple? Maybe I should just try the stable >> >> branch. I will do this next. >> >> >> > >> > I would advise everybody besides about 2 people doing research on I/O >> > scalability with Swift to use the stable branch. Not only does it get >> > fixes before trunk, but it doesn't get weird changes that may cause >> > random breakage. >> > >> >> With the stable branch, and "updated setup" (execution provider >> "local:pbs") I have this error message: >> >> /var/spool/torque/mom_priv/jobs/2489852.abem5.ncsa.uiuc.edu.SC: line >> 10: pdsh: command not found >> >> Should I install pdsh first? > > Yes. Might have a softenv package. > >> ?I didn't see it right away in the TG >> software list. I also don't see instructions in the Swift user guide, >> unless I missed it. > > It's relatively new. There was also the assumption that it would be > installed pretty much everywhere, but it doesn't seem to be the case, so > I', thinking a plain ssh solution (which is what gram does) may be > better. > > From hategan at mcs.anl.gov Mon Feb 1 23:29:54 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 01 Feb 2010 23:29:54 -0600 Subject: [Swift-user] Looking for the cause of failure In-Reply-To: <82f536811002012022g1725800fnd488f88cedb48455@mail.gmail.com> References: <12680489.151301264897544218.JavaMail.root@zimbra> <2795624.151361264897648127.JavaMail.root@zimbra> <82f536811001302007v6940936fn37b9cd79bbab4736@mail.gmail.com> <1264911244.7775.3.camel@localhost> <82f536811001302028p6c58a241l87c4cd9af220c6f8@mail.gmail.com> <1264913153.8312.15.camel@localhost> <82f536811001310749o37a37509i6606f0f6bfcf3be3@mail.gmail.com> <1264953402.11359.1.camel@localhost> <82f536811002012022g1725800fnd488f88cedb48455@mail.gmail.com> Message-ID: <1265088594.22093.2.camel@localhost> On Mon, 2010-02-01 at 23:22 -0500, Andriy Fedorov wrote: > Hi Mihael, > > pdsh does not seem to be available via softenv. I compiled it from > source. My fresh errors are the following: > > "Worker task failed: 0201-560951-000000 Block task ended prematurely" > > and > > "pdsh at abe1177: abe1172: rcmd: socket: Permission denied" > > Do I need to perform some special setup of pdsh? I don't know. I will have to poke around abe and see whether there is a quick fix, but I can't promise anything. > Meanwhile, my > exercises with stick lead me to conclusion that I should rather use > "gt2:gt2:pbs" than "local:pbs" jobmanager in coaster configuration.... I agree. That is probably the best choice. From jamalphd at gmail.com Tue Feb 2 16:13:24 2010 From: jamalphd at gmail.com (J A) Date: Tue, 2 Feb 2010 17:13:24 -0500 Subject: [Swift-user] running unix executables Message-ID: Hi: I am looking for examples that shows how to run UNIX executables using swift. Also, I would to know how to create and delete folders using swift scripts. any help is appreciated. Thanks, Jamal -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Feb 2 16:59:34 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 2 Feb 2010 16:59:34 -0600 (CST) Subject: [Swift-user] running unix executables In-Reply-To: Message-ID: <19255299.229661265151574161.JavaMail.root@zimbra> Jamal, The User Guide and the tutorial on the Swift web have examples of calling executables, eg /bin/echo, /usr/bin/convert, etc. Thats pretty basic - are you reading these materials? For creating directories, etc: you can do this via either a shell script or command you call via an app() function; perhaps by calling /bin/mkdir directly, with site being "localhost". You can also call such a function from an "ext" mapper if you think through the logic carefully. You should read and familiarize yourself with the User Guide material on how Swift jobs run (eg, the "jobdir" sandbox concept), as you'll need to know the difference between a likely-transient dir created below the jobdir vs. a persistent dir created on the local host running the Swift command vs. a persistent dir at a site (which is not likely what you want). - Mike ----- "J A" wrote: > Hi: > > I am looking for examples that shows how to run UNIX executables using > swift. > Also, I would to know how to create and delete folders using swift > scripts. > > any help is appreciated. > > Thanks, > Jamal > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From yizhu at cs.uchicago.edu Wed Feb 3 00:27:28 2010 From: yizhu at cs.uchicago.edu (Yi Zhu) Date: Wed, 03 Feb 2010 00:27:28 -0600 Subject: [Swift-user] two questions about Swift Message-ID: <4B691750.5060909@cs.uchicago.edu> Hi all I have two questions about swift: 1. conventional loop: In swift, we can use foreach to parallel apps, but what If I just need a conventional loop, like a regular "for" or "while" in java? 2. In Matlab, there are some library which support some Math functions like "sin" "cos" "reshape" etc. Is there any similar library available for swift. If not, is there anyway to import external library? (e.g. from Java) Many Thanks. -Yi From zhaozhang at uchicago.edu Wed Feb 3 00:36:25 2010 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Wed, 03 Feb 2010 00:36:25 -0600 Subject: [Swift-user] two questions about Swift In-Reply-To: <4B691750.5060909@cs.uchicago.edu> References: <4B691750.5060909@cs.uchicago.edu> Message-ID: <4B691969.2080705@uchicago.edu> Hi, Yi There is a "Iter" in swift, which could work as a while statement. I am attaching an example. And it is also in swift documentation. http://www.ci.uchicago.edu/swift/guides/tutorial.php#tutorial.iterate Best zhao scipiter (scipin infile, scipopts settings) { scipout ofile[] ; scipin infilearray[] ; scipopts f ; //infilearray[0]=infile; iterate i { ofile[i]=scip(infilearray[i], settings); infilearray[i+1]=parser(ofile[i], f); }until(i==9); } Yi Zhu wrote: > Hi all > > I have two questions about swift: > > 1. conventional loop: In swift, we can use foreach to parallel apps, > but what If I just need a conventional loop, like a regular "for" or > "while" in java? > > 2. In Matlab, there are some library which support some Math > functions like "sin" "cos" "reshape" etc. Is there any similar library > available for swift. If not, is there anyway to import external > library? (e.g. from Java) > > Many Thanks. > > > -Yi > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > From yizhu at cs.uchicago.edu Wed Feb 3 01:05:09 2010 From: yizhu at cs.uchicago.edu (Yi Zhu) Date: Wed, 03 Feb 2010 01:05:09 -0600 Subject: [Swift-user] two questions about Swift In-Reply-To: <4B691969.2080705@uchicago.edu> References: <4B691750.5060909@cs.uchicago.edu> <4B691969.2080705@uchicago.edu> Message-ID: <4B692025.2060600@cs.uchicago.edu> Hi Zhao Thanks for your reply! The "Iterate" seems works like a conventional "while", but it would be great if there also exists a conventional "for" to be nested in foreach loop. -Yi On 2/3/2010 12:36 AM, Zhao Zhang wrote: > Hi, Yi > > There is a "Iter" in swift, which could work as a while statement. I > am attaching an example. > And it is also in swift documentation. > http://www.ci.uchicago.edu/swift/guides/tutorial.php#tutorial.iterate > > Best > zhao > > scipiter (scipin infile, scipopts settings) > { > scipout ofile[] ; > scipin infilearray[] ; > scipopts f ; > //infilearray[0]=infile; > iterate i { > ofile[i]=scip(infilearray[i], settings); > infilearray[i+1]=parser(ofile[i], f); > }until(i==9); > } > > > Yi Zhu wrote: >> Hi all >> >> I have two questions about swift: >> >> 1. conventional loop: In swift, we can use foreach to parallel apps, >> but what If I just need a conventional loop, like a regular "for" or >> "while" in java? >> >> 2. In Matlab, there are some library which support some Math >> functions like "sin" "cos" "reshape" etc. Is there any similar >> library available for swift. If not, is there anyway to import >> external library? (e.g. from Java) >> >> Many Thanks. >> >> >> -Yi >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > From benc at hawaga.org.uk Wed Feb 3 04:30:47 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 3 Feb 2010 10:30:47 +0000 (UTC) Subject: [Swift-user] two questions about Swift In-Reply-To: <4B692025.2060600@cs.uchicago.edu> References: <4B691750.5060909@cs.uchicago.edu> <4B691969.2080705@uchicago.edu> <4B692025.2060600@cs.uchicago.edu> Message-ID: On Wed, 3 Feb 2010, Yi Zhu wrote: > The "Iterate" seems works like a conventional "while", but it would be great > if there also exists a conventional "for" to be nested in foreach loop. What are you trying to do that you can't do with foreach and iterate? -- From fedorov at bwh.harvard.edu Sat Feb 6 12:03:15 2010 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 6 Feb 2010 13:03:15 -0500 Subject: [Swift-user] Specifying execution dependencies directly Message-ID: <82f536811002061003w7680e00dx9cd5808048727761@mail.gmail.com> Hi, I have the following (what seems to me, typical) analysis scenario. I am running multiple instances of task A, task B needs to wait until all of the instances of A are completed, and analyze the outputs produced by As. If I was running this locally, I would pass the name of the directory with the results to B. However, in Swift, I need to somehow define that B cannot start until all of the As complete. Since B would expect a directory name, or an arbitrarily long list of files, I am not sure how this can be done. Should I develop the workflow as a shell script that would run swift script with As, then run B locally, then proceed with the rest of the computation? This seems like a common task in workflows, I would think there is a solution in Swift for this, but I can't seem to find it in the guide. Can anyone help? -- Andriy Fedorov, Ph.D. Research Fellow Brigham and Women's Hospital Harvard Medical School 75 Francis Street Boston, MA 02115 USA fedorov at bwh.harvard.edu From hategan at mcs.anl.gov Sat Feb 6 12:48:41 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 06 Feb 2010 12:48:41 -0600 Subject: [Swift-user] Specifying execution dependencies directly In-Reply-To: <82f536811002061003w7680e00dx9cd5808048727761@mail.gmail.com> References: <82f536811002061003w7680e00dx9cd5808048727761@mail.gmail.com> Message-ID: <1265482121.28632.2.camel@localhost> On Sat, 2010-02-06 at 13:03 -0500, Andriy Fedorov wrote: > Hi, > > I have the following (what seems to me, typical) analysis scenario. I > am running multiple instances of task A, task B needs to wait until > all of the instances of A are completed, and analyze the outputs > produced by As. > > If I was running this locally, I would pass the name of the directory > with the results to B. However, in Swift, I need to somehow define > that B cannot start until all of the As complete. Since B would expect > a directory name, or an arbitrarily long list of files, I am not sure > how this can be done. Should I develop the workflow as a shell script > that would run swift script with As, then run B locally, then proceed > with the rest of the computation? You are right, this is a typical map/reduce problem. Use an array that holds the results of all the As and pass that to B. file aout[]; foreach v, k in someRangeOrSomething { aout[k] = a(v); } file bout; bout = b(aout); From fedorov at bwh.harvard.edu Sat Feb 6 13:20:29 2010 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 6 Feb 2010 14:20:29 -0500 Subject: [Swift-user] Specifying execution dependencies directly In-Reply-To: <1265482121.28632.2.camel@localhost> References: <82f536811002061003w7680e00dx9cd5808048727761@mail.gmail.com> <1265482121.28632.2.camel@localhost> Message-ID: <82f536811002061120k6fdfd20dvfc92dd127749d336@mail.gmail.com> On Sat, Feb 6, 2010 at 13:48, Mihael Hategan wrote: > Use an array that holds the results of all the As and pass that to B. > Yes, that's what I've been looking for. Thanks! > file aout[]; > foreach v, k in someRangeOrSomething { > ?aout[k] = a(v); > } > file bout; > bout = b(aout); > > > From iraicu at cs.uchicago.edu Mon Feb 8 18:25:05 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Mon, 08 Feb 2010 18:25:05 -0600 Subject: [Swift-user] CFP: ACM ScienceCloud 2010 Message-ID: <4B70AB61.7070308@cs.uchicago.edu> CFP reminder: ACM ScienceCloud 2010 This is a kind reminder that the submission deadline for ScienceCloud 2010 is approaching: Event Name: ACM Workshop on Scientific Cloud Computing (ScienceCloud) 2010 Web site: http://dsl.cs.uchicago.edu/ScienceCloud2010/ Venue: High Performance Distributed Computing Symposium (HPDC 2010) Date: June 21st, 2010 Location: Chicago, Illinois, USA Deadlines: Abstract: February 22nd, 2010 Full paper: March 1st, 2010 Acceptance Notification: April 1st, 2010 Camera Ready Papers: April 15th, 2010 -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= From mmaiensc at uchicago.edu Fri Feb 12 17:18:53 2010 From: mmaiensc at uchicago.edu (Mark Maienschein-Cline) Date: Fri, 12 Feb 2010 17:18:53 -0600 Subject: [Swift-user] Error using programs in swift Message-ID: <9927869C-1437-45AA-B53C-6A1F22E1A854@uchicago.edu> Hi, I'm getting a "java.lang.NullPointerException" error when I call some of my programs (written in C) in Swift (in the local mode). The programs work correctly when I run them from the command line, and I don't think there is any error in the parameters swift is passing to the program. Does anyone know what problems can cause this kind of error? Thanks, Mark From wilde at mcs.anl.gov Fri Feb 12 17:47:54 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 12 Feb 2010 17:47:54 -0600 (CST) Subject: [Swift-user] Error using programs in swift In-Reply-To: <9927869C-1437-45AA-B53C-6A1F22E1A854@uchicago.edu> Message-ID: <30064923.551521266018474313.JavaMail.root@zimbra> Mark, a null ptr exception is always an internal Swift error triggered by some problem in Java. Please send to swift-devel#ci.uchicago.edu the Swift script and the Swift log file from the run (the one called some-long-name.log) - or tell us where we can find it on the CI network. Thanks, Mike ----- "Mark Maienschein-Cline" wrote: > Hi, > > I'm getting a "java.lang.NullPointerException" error when I call some > of my programs (written in C) in Swift (in the local mode). The > programs work correctly when I run them from the command line, and I > don't think there is any error in the parameters swift is passing to > the program. Does anyone know what problems can cause this kind of > error? > > Thanks, > Mark_______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From wilde at mcs.anl.gov Tue Feb 16 11:12:14 2010 From: wilde at mcs.anl.gov (wilde at mcs.anl.gov) Date: Tue, 16 Feb 2010 11:12:14 -0600 (CST) Subject: [Swift-user] two questions about Swift In-Reply-To: <9904473.616281266340313500.JavaMail.root@zimbra> Message-ID: <9743673.616301266340334260.JavaMail.root@zimbra> Yi, sorry - I missed these questions from you; just saw them now. ----- "Yi Zhu" wrote: > Hi all > > I have two questions about swift: > > 1. conventional loop: In swift, we can use foreach to parallel apps, > > but what If I just need a conventional loop, like a regular "for" or > "while" in java? No, thats somewhat limited at the moment. We can consider this for future revisions. There were a variety of differing views on what control statements Swift should offer. > > 2. In Matlab, there are some library which support some Math > functions > like "sin" "cos" "reshape" etc. Is there any similar library available > > for swift. If not, is there anyway to import external library? (e.g. > from Java) I have experimented with techniques for building a library of useful functions by using simple app() functions that do their work using a variety of shell commands and scripts. Pasted below is one of my experiments; Im not sure if all the functions in it work, but it should serve as a guide for *one* way of doing this. At the same time, a guide for extending the set of built-in primitives would be useful as well, as these can bend the data typing rules in useful ways, whereas the app() approach to building lib functions can not (so you often wind up with a lot fo explicit "casting" functions to e.g. jump between numbers and strings). - Mike --- login1$ cat stringlib.swift type file; // Lib functions app (file o) echoi (int i) { echo i stdout=@o;} app (file o) echof (float f) { echo f stdout=@o;} app (file o) echob (boolean b) { echo b stdout=@o;} app (file o) echos (string s) { echo s stdout=@o;} (string s) itostr (int i) { file f; f = echoi(i); s = readData(f); } (string s) ftostr (float n) { file f; f = echof(n); s = readData(f); } (float n) strtof (string s) { file f; f = echos(s); n = readData(f); } app (file o) sprintfsApp (string fmt, string e[]) { sprintfs fmt e stdout=@o; } (string s) sprintfs (string fmt, string e[]) { file f; f = sprintfsApp(fmt,e); s = readData(f); } // Tests test_sprintf() { string s0 = sprintfs("%.10d",["123"]); trace(s0); string s1 = sprintfs("input/protein.%s.%.10d",["T1ubq",itostr(123)]); trace(s1); string s2 = sprintfs("n=%.4e",[ftostr(3.34e06)]); trace(s2); } test_converters() { float f = strtof("123.456"); trace("strtof(123.456)", f); } test_converters(); login1$ swift stringlib.swift Swift svn swift-r3202 cog-r2683 RunID: 20100216-1108-mg4uz0v7 Progress: SwiftScript trace: strtof(123.456), 123.456 Final status: Finished successfully:1 login1$ > > Many Thanks. > > > -Yi > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From HodgessE at uhd.edu Tue Feb 16 11:42:16 2010 From: HodgessE at uhd.edu (Hodgess, Erin) Date: Tue, 16 Feb 2010 11:42:16 -0600 Subject: [Swift-user] two questions about Swift References: <9743673.616301266340334260.JavaMail.root@zimbra> Message-ID: <101CBF5360343B45B9C9B0B05D71474C25B110@BALI.uhd.campus> You can use R, the stat program. That works with Swift. There is a tutorial that Mike can point you to. Thanks, Erin Erin M. Hodgess, PhD Associate Professor Department of Computer and Mathematical Sciences University of Houston - Downtown mailto: hodgesse at uhd.edu -----Original Message----- From: swift-user-bounces at ci.uchicago.edu on behalf of wilde at mcs.anl.gov Sent: Tue 2/16/2010 11:12 AM To: Yi Zhu Cc: swift-user at ci.uchicago.edu Subject: Re: [Swift-user] two questions about Swift Yi, sorry - I missed these questions from you; just saw them now. ----- "Yi Zhu" wrote: > Hi all > > I have two questions about swift: > > 1. conventional loop: In swift, we can use foreach to parallel apps, > > but what If I just need a conventional loop, like a regular "for" or > "while" in java? No, thats somewhat limited at the moment. We can consider this for future revisions. There were a variety of differing views on what control statements Swift should offer. > > 2. In Matlab, there are some library which support some Math > functions > like "sin" "cos" "reshape" etc. Is there any similar library available > > for swift. If not, is there anyway to import external library? (e.g. > from Java) I have experimented with techniques for building a library of useful functions by using simple app() functions that do their work using a variety of shell commands and scripts. Pasted below is one of my experiments; Im not sure if all the functions in it work, but it should serve as a guide for *one* way of doing this. At the same time, a guide for extending the set of built-in primitives would be useful as well, as these can bend the data typing rules in useful ways, whereas the app() approach to building lib functions can not (so you often wind up with a lot fo explicit "casting" functions to e.g. jump between numbers and strings). - Mike --- login1$ cat stringlib.swift type file; // Lib functions app (file o) echoi (int i) { echo i stdout=@o;} app (file o) echof (float f) { echo f stdout=@o;} app (file o) echob (boolean b) { echo b stdout=@o;} app (file o) echos (string s) { echo s stdout=@o;} (string s) itostr (int i) { file f; f = echoi(i); s = readData(f); } (string s) ftostr (float n) { file f; f = echof(n); s = readData(f); } (float n) strtof (string s) { file f; f = echos(s); n = readData(f); } app (file o) sprintfsApp (string fmt, string e[]) { sprintfs fmt e stdout=@o; } (string s) sprintfs (string fmt, string e[]) { file f; f = sprintfsApp(fmt,e); s = readData(f); } // Tests test_sprintf() { string s0 = sprintfs("%.10d",["123"]); trace(s0); string s1 = sprintfs("input/protein.%s.%.10d",["T1ubq",itostr(123)]); trace(s1); string s2 = sprintfs("n=%.4e",[ftostr(3.34e06)]); trace(s2); } test_converters() { float f = strtof("123.456"); trace("strtof(123.456)", f); } test_converters(); login1$ swift stringlib.swift Swift svn swift-r3202 cog-r2683 RunID: 20100216-1108-mg4uz0v7 Progress: SwiftScript trace: strtof(123.456), 123.456 Final status: Finished successfully:1 login1$ > > Many Thanks. > > > -Yi > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Tue Feb 16 14:22:25 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Tue, 16 Feb 2010 14:22:25 -0600 (CST) Subject: [Swift-user] two questions about Swift In-Reply-To: <8812436.625921266350733766.JavaMail.root@zimbra> Message-ID: <12220671.626861266351745204.JavaMail.root@zimbra> ----- "Justin M Wozniak" wrote: > I can quickly deliver sin() and cos() by adding to swiftscript.Misc. > I > definitely don't think he should call R for sin(). However: > > 1) What if we supported: > > x = 0.5; > library("L", "org.my.lib"); > L:sin(x); > > where org.my.lib is in the classpath and extends FunctionsCollection? > I > think this might be easily doable. That sounds pretty useful for general extensibility. I'd also suggest (in addition) a similar extension interface, but which like the "ext" mapper lets you write Swift functions as shell scripts or arbitrary executables. Note that you can do sin() and basic math functions in awk or bc, so the app() based extensions would be a good way to prototype things initially. > > 2) Is there any way to get to the Karajan library? I don't know. Mihael or Ben? - Mike > > On Tue, 16 Feb 2010, Hodgess, Erin wrote: > > > You can use R, the stat program. That works with Swift. > > > > There is a tutorial that Mike can point you to. > > > > Thanks, > > Erin > > > > > > Erin M. Hodgess, PhD > > Associate Professor > > Department of Computer and Mathematical Sciences > > University of Houston - Downtown > > mailto: hodgesse at uhd.edu > > > > > > > > -----Original Message----- > > From: swift-user-bounces at ci.uchicago.edu on behalf of > wilde at mcs.anl.gov > > Sent: Tue 2/16/2010 11:12 AM > > To: Yi Zhu > > Cc: swift-user at ci.uchicago.edu > > Subject: Re: [Swift-user] two questions about Swift > > > > Yi, sorry - I missed these questions from you; just saw them now. > > > > ----- "Yi Zhu" wrote: > > > >> Hi all > >> > >> 2. In Matlab, there are some library which support some Math > functions > >> like "sin" "cos" "reshape" etc. Is there any similar library > available > >> for swift. If not, is there anyway to import external library? > (e.g. > >> from Java) > > -- > Justin M Wozniak From benc at hawaga.org.uk Tue Feb 16 16:06:54 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Tue, 16 Feb 2010 22:06:54 +0000 (GMT) Subject: [Swift-user] two questions about Swift In-Reply-To: <12220671.626861266351745204.JavaMail.root@zimbra> References: <12220671.626861266351745204.JavaMail.root@zimbra> Message-ID: > I'd also suggest (in addition) a similar extension interface, but which > like the "ext" mapper lets you write Swift functions as shell scripts or > arbitrary executables. I think anywhere you would use a function you can also call a procedure, in which case you could call an app procedure. I can't remember if that actually works in all cases. -- From tiberius at ci.uchicago.edu Wed Feb 17 10:48:38 2010 From: tiberius at ci.uchicago.edu (Tiberiu Stef-Praun) Date: Wed, 17 Feb 2010 10:48:38 -0600 Subject: [Swift-user] Problem with iterate Message-ID: Hi Guys I have put together this really simple iteration example, which I expected to work (and it used to work in the past): ============== type file; app (file initOut) initFunc (string inputString){ echo inputString stdout=@filename(initOut); } app (file catOut) catFunc (file catIn){ cat @filename(catIn) @filename(catOut); } runLoop(){ file statePathFiles[]; statePathFiles[0]=initFunc("hello"); iterate it{ trace(@strcat("Iteration: ",it)); statePathFiles[it+1]=catFunc(statePathFiles[it]); } until (it==0); } runLoop(); ================ However, I get an error like this: Running UC Eval Could not start execution. variable statePathFiles has multiple writers. Please suggest solutions for it . Thank you Tibi -- Tiberiu (Tibi) Stef-Praun, PhD Computational Sciences Researcher Computation Institute 5640 S. Ellis Ave, #405 University of Chicago http://www-unix.mcs.anl.gov/~tiberius/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Wed Feb 17 10:56:33 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Wed, 17 Feb 2010 10:56:33 -0600 (CST) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: Message-ID: <21055881.656131266425793594.JavaMail.root@zimbra> Tibi, while it may not be the most elegant approach, I think you need to put all statements that set the array statePathFiles *inside* the loop, and conditionally execute the setting of element 0 using an if statement. - Mike ----- "Tiberiu Stef-Praun" wrote: > Hi Guys > > I have put together this really simple iteration example, which I > expected to work (and it used to work in the past): > > ============== > type file; > > app (file initOut) initFunc (string inputString){ > echo inputString stdout=@filename(initOut); > } > > app (file catOut) catFunc (file catIn){ > cat @filename(catIn) @filename(catOut); > } > > runLoop(){ > > file statePathFiles[] suffix=".mat">; > > statePathFiles[0]=initFunc("hello"); > > iterate it{ > trace(@strcat("Iteration: ",it)); > statePathFiles[it+1]=catFunc(statePathFiles[it]); > } until (it==0); > } > > runLoop(); > ================ > > However, I get an error like this: > > Running UC Eval > Could not start execution. > variable statePathFiles has multiple writers. > > > Please suggest solutions for it . > > Thank you > Tibi > > > > -- > Tiberiu (Tibi) Stef-Praun, PhD > Computational Sciences Researcher > Computation Institute > 5640 S. Ellis Ave, #405 > University of Chicago > http://www-unix.mcs.anl.gov/~tiberius/ > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel From benc at hawaga.org.uk Wed Feb 17 10:58:42 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Wed, 17 Feb 2010 16:58:42 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: Message-ID: > I have put together this really simple iteration example, which I expected > to work (and it used to work in the past): mmm. yeah, the static analysis of when array elements are allowed to be assigned doesn't work well there. If anyone is looking at fixing it, I think its something like the iterate statement being tagged as a writer of "all of statePathFiles", which then conflicts with the first line being a partial writer of statePathFiles. If anyone is *really* look at fixing it, get rid of foreach and iterate and put in proper dataflow based iteration (map, foldr, whatever) such that arrays are single-assignment. > statePathFiles[0]=initFunc("hello"); > > iterate it{ > trace(@strcat("Iteration: ",it)); > statePathFiles[it+1]=catFunc(statePathFiles[it]); > } until (it==0); > } > Please suggest solutions for it . probably the two solutions above are not what you're looking for. but: move the statePathFiles[0] assignment into the iterate loop, by saying something like the below (but you'll have to adjust the array indices too, I think, because you need to start one earlier now). something like this: iterate it { if(it==0) { statePathFiles[0]=initFunc("hello"); } else {statePathFiles[it+1]=catFunc(statePathFiles[it]);} } until(it==0); -- From wilde at mcs.anl.gov Wed Feb 17 11:05:21 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Wed, 17 Feb 2010 11:05:21 -0600 (CST) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: Message-ID: <12659274.656851266426321107.JavaMail.root@zimbra> An example of the "if 0" approach, Tibi: iterate r { if ( r==0 ) { pdata[0] = ItFixInit(p); } else { sim[r] = doRound(p, nsim, config, "runoops"); (pdata[r], converged[r]) = analyze(sim[r], pdata[r-1], r, maxr); } } until ( r==maxr ); Ben, thanks for the suggested fix. - Mike ----- "Ben Clifford" wrote: > > I have put together this really simple iteration example, which I > expected > > to work (and it used to work in the past): > > mmm. > > yeah, the static analysis of when array elements are allowed to be > assigned doesn't work well there. > > If anyone is looking at fixing it, I think its something like the > iterate > statement being tagged as a writer of "all of statePathFiles", which > then > conflicts with the first line being a partial writer of > statePathFiles. > > If anyone is *really* look at fixing it, get rid of foreach and > iterate > and put in proper dataflow based iteration (map, foldr, whatever) such > > that arrays are single-assignment. > > > statePathFiles[0]=initFunc("hello"); > > > > iterate it{ > > trace(@strcat("Iteration: ",it)); > > statePathFiles[it+1]=catFunc(statePathFiles[it]); > > } until (it==0); > > } > > > Please suggest solutions for it . > > probably the two solutions above are not what you're looking for. > > but: > > move the statePathFiles[0] assignment into the iterate loop, by saying > > something like the below (but you'll have to adjust the array indices > too, > I think, because you need to start one earlier now). something like > this: > > iterate it { > if(it==0) { statePathFiles[0]=initFunc("hello"); } else > {statePathFiles[it+1]=catFunc(statePathFiles[it]);} > } until(it==0); > > > -- > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From hategan at mcs.anl.gov Wed Feb 17 13:24:25 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 17 Feb 2010 13:24:25 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: Message-ID: <1266434665.9702.6.camel@localhost> On Wed, 2010-02-17 at 16:58 +0000, Ben Clifford wrote: > If anyone is *really* look at fixing it, get rid of foreach and iterate > and put in proper dataflow based iteration (map, foldr, whatever) such > that arrays are single-assignment. Even in functional languages some of these problems are solved by recursion rather than a single pre-defined function. Which may or may not be fine as a usability issue, but it's almost clearly not fine due to the lack of tail recursion optimization in swift. From nbest at ci.uchicago.edu Wed Feb 17 14:11:07 2010 From: nbest at ci.uchicago.edu (Neil Best) Date: Wed, 17 Feb 2010 14:11:07 -0600 Subject: [Swift-user] two questions about Swift In-Reply-To: <12220671.626861266351745204.JavaMail.root@zimbra> References: <12220671.626861266351745204.JavaMail.root@zimbra> Message-ID: On Feb 16, 2010, at 2:22 PM, Michael Wilde wrote: > Note that you can do sin() and basic math functions in awk or bc, so > the app() based extensions would be a good way to prototype things > initially. This might also be useful: http://dirk.eddelbuettel.com/code/littler.html > littler: a scripting front-end for GNU R > > littler (pronounced 'little R' and written 'r') provides hash-bang > (i.e. script starting with #!/some/path) capability for GNU R, as > well as simple command-line and piping use. From hategan at mcs.anl.gov Wed Feb 17 16:17:08 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 17 Feb 2010 16:17:08 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: Message-ID: <1266445028.16024.13.camel@localhost> I committed a bunch of patches to trunk... > > If anyone is looking at fixing it, I think its something like the iterate > statement being tagged as a writer of "all of statePathFiles", which then > conflicts with the first line being a partial writer of statePathFiles. That one. I'm not sure about the implications of it. Why was that flag set to "not partial writer"? How did iterate ever work? What am I missing? > > If anyone is *really* look at fixing it, get rid of foreach and iterate > and put in proper dataflow based iteration (map, foldr, whatever) such > that arrays are single-assignment. There's also a bunch of fixes that should enable the use of a normal foreach for the same purpose. Basically the code detects if iteration happens on a variable that is updated in the body of the foreach and sets a special flag on the foreach when that happens. The foreach implementation then interprets (with some nasty details about the concurrency involved which I hopefully got right) situations with no running iterations as a sign that things are done. So the following works as expected: int a[]; a[0] = 50; foreach v, k in a { trace(v); if (v > 0) { a[k + 1] = v - 1; } } Unfortunately static analysis fails to detect more complex patterns, such as when there is an indirect dependence between the iteration variable and the array that the body writes to. So the following deadlocks: int a[]; int b[]; a[0] = 50; foreach v, k in a { trace(v); if (v > 0) { b[k + 1] = v - 1; } } foreach v, k in b { a[k] = b[k]; } Ideas? From benc at hawaga.org.uk Thu Feb 18 02:46:17 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 18 Feb 2010 08:46:17 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266445028.16024.13.camel@localhost> References: <1266445028.16024.13.camel@localhost> Message-ID: > Why was that flag > set to "not partial writer"? I think that was a bug rather than some deep reason. > How did iterate ever work? What am I missing? I think it may have only ever worked in the initialise-in-nested-if case (or fairly trivial test cases which don't run into this problem) > Ideas? I'm split between two minds: a) more static and runtime analysis can lead to some (informally provable?) correct behaviour here. It think it might be possible to at least detect deadlocks caused by this and more elegantly error out, by finding cycles in a graph of who is waiting for what, which is a better user experience than a deadlock (as in, you can type the deadlock message into google and get the beautiful documentation page all about resolving such problems) b) that having multiple writer arrays and foreach/iterate control structures in their present for are (informally provably?) inherently deadlock prone and no matter what is done, it will always be possible to contrive a deadlock; and so then to have a "correct" system, those structures need to fundamentally change. Even if b is the case, it might be that problems are sufficiently obscure that it doesn't matter from the "lets run a bazillion protein folds" perspective. -- From hategan at mcs.anl.gov Thu Feb 18 11:04:49 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Feb 2010 11:04:49 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> Message-ID: <1266512689.6317.30.camel@localhost> On Thu, 2010-02-18 at 08:46 +0000, Ben Clifford wrote: > > Why was that flag > > set to "not partial writer"? > > I think that was a bug rather than some deep reason. > > > How did iterate ever work? What am I missing? > > I think it may have only ever worked in the initialise-in-nested-if case > (or fairly trivial test cases which don't run into this problem) > > > Ideas? > > I'm split between two minds: > > a) more static and runtime analysis can lead to some (informally > provable?) correct behaviour here. It think it might be possible to at > least detect deadlocks caused by this and more elegantly error out, by > finding cycles in a graph of who is waiting for what, which is a > better user experience than a deadlock (as in, you can type the > deadlock message into google and get the beautiful documentation page > all about resolving such problems) > > b) that having multiple writer arrays and foreach/iterate control > structures in their present for are (informally provably?) inherently > deadlock prone and no matter what is done, it will always be possible > to contrive a deadlock; and so then to have a "correct" system, those > structures need to fundamentally change. I'm unsure. I have a feeling that static dealdock detection is undecidable in swift. I need to think some more about that. > > Even if b is the case, it might be that problems are sufficiently obscure > that it doesn't matter from the "lets run a bazillion protein folds" > perspective. > Right. Which is what we've been doing so far. From benc at hawaga.org.uk Thu Feb 18 11:08:59 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 18 Feb 2010 17:08:59 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266512689.6317.30.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> Message-ID: > I'm unsure. I have a feeling that static dealdock detection is > undecidable in swift. I need to think some more about that. I think doing it statically (i.e. at compile time) is equivalent to executing the code (including external apps) - I think there's probably a straightforward example where something depends on an integer read from an externally executed app which then influences an if statement that decides which of two different arrays is being written to; or something like that. But I think it could be done at runtime (which is less pleasing, but still better than a runtime hang). -- From hategan at mcs.anl.gov Thu Feb 18 16:07:29 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Feb 2010 16:07:29 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> Message-ID: <1266530849.15179.107.camel@localhost> On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: > > I'm unsure. I have a feeling that static dealdock detection is > > undecidable in swift. I need to think some more about that. > > I think doing it statically (i.e. at compile time) is equivalent to > executing the code (including external apps) - I think there's probably a > straightforward example where something depends on an integer read from an > externally executed app which then influences an if statement that decides > which of two different arrays is being written to; or something like that. Right. So termination is undecidable by virtue of it being a turing complete language. What I'm unsure is whether deadlocks follow as a consequence of that. They are a different beast. We can probably start with simpler things. The following deadlocks with the current swift: a = cat(b); b = cat(a); So it seems that loops in the graph will create a deadlock and detecting loops requires knowing the graph. We can probably detect the static parts and we may be able to handle alternatives by issuing warnings of the "branch 1 in that 'if' combined with branch 2 in that other 'if' leads to a deadlock" sort. Arrays may be a bit tougher. From iraicu at cs.uchicago.edu Thu Feb 18 16:17:22 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Thu, 18 Feb 2010 16:17:22 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266530849.15179.107.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> Message-ID: <4B7DBC72.6020403@cs.uchicago.edu> But wouldn't the single assignment part prevent your example from dead-locking? In other words, the compiler should throw an error on line "b = cat(a)" as its trying to modify an existing value b. Mihael Hategan wrote: > On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: > >>> I'm unsure. I have a feeling that static dealdock detection is >>> undecidable in swift. I need to think some more about that. >>> >> I think doing it statically (i.e. at compile time) is equivalent to >> executing the code (including external apps) - I think there's probably a >> straightforward example where something depends on an integer read from an >> externally executed app which then influences an if statement that decides >> which of two different arrays is being written to; or something like that. >> > > Right. So termination is undecidable by virtue of it being a turing > complete language. What I'm unsure is whether deadlocks follow as a > consequence of that. They are a different beast. > > We can probably start with simpler things. The following deadlocks with > the current swift: > a = cat(b); > b = cat(a); > > So it seems that loops in the graph will create a deadlock and detecting > loops requires knowing the graph. We can probably detect the static > parts and we may be able to handle alternatives by issuing warnings of > the "branch 1 in that 'if' combined with branch 2 in that other 'if' > leads to a deadlock" sort. > > Arrays may be a bit tougher. > > > _______________________________________________ > Swift-devel mailing list > Swift-devel at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Feb 18 16:37:43 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Feb 2010 16:37:43 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7DBC72.6020403@cs.uchicago.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> Message-ID: <1266532663.17633.0.camel@localhost> There is no existing value of b. Here's the full example, which you can try out: app (file y) cat(file x) { cat @x stdout=@y; } type file; file a; file b; a = cat(b); b = cat(a); On Thu, 2010-02-18 at 16:17 -0600, Ioan Raicu wrote: > But wouldn't the single assignment part prevent your example from > dead-locking? In other words, the compiler should throw an error on > line "b = cat(a)" as its trying to modify an existing value b. > > Mihael Hategan wrote: > > On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: > > > > > > I'm unsure. I have a feeling that static dealdock detection is > > > > undecidable in swift. I need to think some more about that. > > > > > > > I think doing it statically (i.e. at compile time) is equivalent to > > > executing the code (including external apps) - I think there's probably a > > > straightforward example where something depends on an integer read from an > > > externally executed app which then influences an if statement that decides > > > which of two different arrays is being written to; or something like that. > > > > > > > Right. So termination is undecidable by virtue of it being a turing > > complete language. What I'm unsure is whether deadlocks follow as a > > consequence of that. They are a different beast. > > > > We can probably start with simpler things. The following deadlocks with > > the current swift: > > a = cat(b); > > b = cat(a); > > > > So it seems that loops in the graph will create a deadlock and detecting > > loops requires knowing the graph. We can probably detect the static > > parts and we may be able to handle alternatives by issuing warnings of > > the "branch 1 in that 'if' combined with branch 2 in that other 'if' > > leads to a deadlock" sort. > > > > Arrays may be a bit tougher. > > > > > > _______________________________________________ > > Swift-devel mailing list > > Swift-devel at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel > > > > > > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > From benc at hawaga.org.uk Thu Feb 18 17:19:14 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Thu, 18 Feb 2010 23:19:14 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7DC3DF.4070107@eecs.northwestern.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> Message-ID: > But I don't get it, how can the variable b not have a value, if it is consumed > by cat() in a prior statement? Its not prior in execution order; its prior in source text, which does not imply execution order. If I write (in maths, not in a program) this system of simultaneous equations: x = y + 1 y = x + 1 then there is no solution giving values for x and y. Whats happening there is fairly close to what Mihael's example was (but saying +1 instead of cat, and using integers). > I thought Swift followed a write-once read many > paradigm, which was enforced by the single assignment paradigm. yes (apart from arrays, which are monotonically assigned so have most of the same properties as single assignment variables) > I guess there is a > distinction between files and variables? Variables hold "things". Those things can be in-memory values, or they can be files. But for this discussion the distinction doesn't matter - the file/in-memory value stuff is orthogonal to the data-directed execution. -- From hategan at mcs.anl.gov Thu Feb 18 20:13:29 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 18 Feb 2010 20:13:29 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> Message-ID: <1266545609.17911.3.camel@localhost> On Thu, 2010-02-18 at 23:19 +0000, Ben Clifford wrote: > > > But I don't get it, how can the variable b not have a value, if it is consumed > > by cat() in a prior statement? > > Its not prior in execution order; its prior in source text, which does not > imply execution order. > > If I write (in maths, not in a program) this system of simultaneous > equations: > > x = y + 1 > y = x + 1 A formulation which beautifully captures the nature of a Swift script. "A set of simultaneous equations". From benc at hawaga.org.uk Fri Feb 19 05:13:55 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 19 Feb 2010 11:13:55 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266530849.15179.107.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> Message-ID: > Right. So termination is undecidable by virtue of it being a turing > complete language. What I'm unsure is whether deadlocks follow as a > consequence of that. They are a different beast. Turing complete languages don't deadlock in general. (yes, concurrent programs do, but regard swift as a serial programming language, not as a concurrent one; and regard concurrent execution as 'merely' an execution optimisation) So I don't think deadlocks are a consequence of that. But I do think that probably finding deadlocks given the language features of SwiftScript is undecideable (not as a consequence of turing completeness, but because of our choice of language features); and on a more practical note, requires knowledge of the output of app blocks. -- From iraicu at cs.uchicago.edu Fri Feb 19 10:48:56 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Fri, 19 Feb 2010 10:48:56 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> Message-ID: <4B7EC0F8.7060005@cs.uchicago.edu> But assuming there are dependencies, why wouldn't the order in source code also imply order in execution order? Mathematically, the two different sequences evaluate to different values: x = y + 1 y = x + 1 assuming y = 0, x = 1 you get: x = 0 + 1 = 1 y = 1 + 1 = 2 final values for x = 1, y = 2 Now if the order was reversed: y = x + 1 x = y + 1 assuming the same initial values y = 0, x = 1 y = 1 + 1 = 2 x = 2 + 1 = 3 Notice that the final value for x is different, depending on the order of the execution. So, since the order matters (because there are dependencies between the 2 statements, as if there were no dependencies the order would be irrelevant), then it makes sense that the execution order also follow the source code order, as the programmer likely wrote the code thinking that this order would be preserved. So, I ask again, why would the execution order not follow the source code order, assuming there was some dependency among them? Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Ben Clifford wrote: > >> But I don't get it, how can the variable b not have a value, if it is consumed >> by cat() in a prior statement? >> > > Its not prior in execution order; its prior in source text, which does not > imply execution order. > > If I write (in maths, not in a program) this system of simultaneous > equations: > > x = y + 1 > y = x + 1 > > then there is no solution giving values for x and y. Whats happening there > is fairly close to what Mihael's example was (but saying +1 instead of > cat, and using integers). > > >> I thought Swift followed a write-once read many >> paradigm, which was enforced by the single assignment paradigm. >> > > yes (apart from arrays, which are monotonically assigned so have most of > the same properties as single assignment variables) > > >> I guess there is a >> distinction between files and variables? >> > > Variables hold "things". Those things can be in-memory values, or they can > be files. But for this discussion the distinction doesn't matter - the > file/in-memory value stuff is orthogonal to the data-directed execution. > > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From benc at hawaga.org.uk Fri Feb 19 10:51:39 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 19 Feb 2010 16:51:39 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7EC0F8.7060005@cs.uchicago.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> Message-ID: > Mathematically, the two different sequences evaluate to different values: > > x = y + 1 > y = x + 1 > > assuming y = 0, x = 1 I mean in simultaneous equations (linear algebra) - in other words, "find (through whatever means you care to use) a value of x and y such that the above two equations are both satisfied" - there is no value of x and y that satisfies that. -- From iraicu at cs.uchicago.edu Fri Feb 19 11:01:55 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Fri, 19 Feb 2010 11:01:55 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> Message-ID: <4B7EC403.9090307@cs.uchicago.edu> Sent again from my UChicago adress: Ioan Raicu wrote: > But lets bring this back to a more real example. A user wanting to > express some computations that have some dependencies, would write out > their computations in some order, expecting their order to be > preserved because of the dependencies. If you only support single > assignment on variables (e.g. the data), then an example like the one > below could never deadlock because the single assignment would be > violated on the 2nd statement. Perhaps things are more complicated if > you support multiple assignments per variables, but that is not the > case for Swift, right? > > I am trying to understand if this deadlock is happening in Swift due > to some particular implementation detail in Swift (or underlying > pieces), or is it a fundamental flaw in the DAG based approach with > single assignment variables? Or is it due to something completely > different? > > Thanks, > Ioan > -- > ================================================================= > Ioan Raicu, Ph.D. > NSF/CRA Computing Innovation Fellow > ================================================================= > Center for Ultra-scale Computing and Information Security (CUCIS) > Department of Electrical Engineering and Computer Science > Northwestern University > 2145 Sheridan Rd, Tech M384 > Evanston, IL 60208-3118 > ================================================================= > Cel: 1-847-722-0876 > Tel: 1-847-491-8163 > Email: iraicu at eecs.northwestern.edu > Web: http://www.eecs.northwestern.edu/~iraicu/ > https://wiki.cucis.eecs.northwestern.edu/ > ================================================================= > ================================================================= > > > > > Ben Clifford wrote: >>> Mathematically, the two different sequences evaluate to different values: >>> >>> x = y + 1 >>> y = x + 1 >>> >>> assuming y = 0, x = 1 >>> >> >> I mean in simultaneous equations (linear algebra) - in other words, "find >> (through whatever means you care to use) a value of x and y such that the >> above two equations are both satisfied" - there is no value of x and y >> that satisfies that. >> >> -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Fri Feb 19 11:34:39 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 19 Feb 2010 11:34:39 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> Message-ID: <1266600879.22169.277.camel@localhost> On Fri, 2010-02-19 at 10:59 -0600, Ioan Raicu wrote: > But lets bring this back to a more real example. A user wanting to > express some computations that have some dependencies, would write out > their computations in some order, expecting their order to be > preserved because of the dependencies. If you only support single > assignment on variables (e.g. the data), then an example like the one > below could never deadlock because the single assignment would be > violated on the 2nd statement. Perhaps things are more complicated if > you support multiple assignments per variables, but that is not the > case for Swift, right? x = y + 1 y = x + 1 How many assignments are there? 2 How many variables are being assigned? 2 Do those assignments have the same lvalue (thing on the left)? no --- Conclusion: each variable is assigned exactly once. > > I am trying to understand if this deadlock is happening in Swift due > to some particular implementation detail in Swift (or underlying > pieces), or is it a fundamental flaw in the DAG based approach with > single assignment variables? Or is it due to something completely > different? It seems to be a fundamental flaw with computers and algorithms. From iraicu at cs.uchicago.edu Fri Feb 19 11:41:37 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Fri, 19 Feb 2010 11:41:37 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266600879.22169.277.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> <1266600879.22169.277.camel@localhost> Message-ID: <4B7ECD51.9040005@cs.uchicago.edu> Mihael Hategan wrote: > On Fri, 2010-02-19 at 10:59 -0600, Ioan Raicu wrote: > >> But lets bring this back to a more real example. A user wanting to >> express some computations that have some dependencies, would write out >> their computations in some order, expecting their order to be >> preserved because of the dependencies. If you only support single >> assignment on variables (e.g. the data), then an example like the one >> below could never deadlock because the single assignment would be >> violated on the 2nd statement. Perhaps things are more complicated if >> you support multiple assignments per variables, but that is not the >> case for Swift, right? >> > > x = y + 1 > y = x + 1 > > How many assignments are there? 2 > How many variables are being assigned? 2 > Do those assignments have the same lvalue (thing on the left)? no > --- > Conclusion: each variable is assigned exactly once. > However, y and x have to be defined somewhere, right? If they don't get their value assigned at definition time, then the statement x = y + 1 should return with an error that y is not defined. > >> I am trying to understand if this deadlock is happening in Swift due >> to some particular implementation detail in Swift (or underlying >> pieces), or is it a fundamental flaw in the DAG based approach with >> single assignment variables? Or is it due to something completely >> different? >> > > It seems to be a fundamental flaw with computers and algorithms. > -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From benc at hawaga.org.uk Fri Feb 19 11:54:28 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Fri, 19 Feb 2010 17:54:28 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <4B7EC377.1030105@eecs.northwestern.edu> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> Message-ID: A point of information related to this discussion: Underordered single assignment doesn't necessarily lead to deadlock in general. Using Haskell, and making this mutually recursive definition of x and y. f l = 1 : l -- if you like lisp, 1:l means (cons 1 l) x = f y y = f x main = print x x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so running the above code prints out a never-ending (until ctrl-c) list of 1,1,1,1,...) Also in Haskell, if you substitute in this, f l = if length l < 200 then 1 : l else l then Haskell detects a loop at runtime (not compile time) - that's pretty much the kind of "deadlock detection" that we're talking about here. $ ./example2 example2: <> (or indeed anything involving the use of 'length l' I think) It might be reasonable to not attempt to do something in this space if ghc cannot do it. Now whats different between the above example and swiftscript? well, I used lists which is a data type that Swift doesn't have. Though maybe its possible to make something like this using arrays/structs somehow. -- http://www.hawaga.org.uk/ben/ From hategan at mcs.anl.gov Fri Feb 19 15:01:16 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 19 Feb 2010 15:01:16 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> Message-ID: <1266613276.31696.1.camel@localhost> On Fri, 2010-02-19 at 17:54 +0000, Ben Clifford wrote: > A point of information related to this discussion: > > Underordered single assignment doesn't necessarily lead to deadlock in > general. Using Haskell, and making this mutually recursive definition of x > and y. > > f l = 1 : l -- if you like lisp, 1:l means (cons 1 l) > > x = f y > y = f x > > main = print x > > x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so > running the above code prints out a never-ending (until ctrl-c) list of > 1,1,1,1,...) What difference does that make? To the outside observer both programs hang. From hategan at mcs.anl.gov Fri Feb 19 19:06:11 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 19 Feb 2010 19:06:11 -0600 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> Message-ID: <1266627971.11280.1.camel@localhost> On Fri, 2010-02-19 at 17:54 +0000, Ben Clifford wrote: > A point of information related to this discussion: > > Underordered single assignment doesn't necessarily lead to deadlock in > general. Using Haskell, and making this mutually recursive definition of x > and y. > > f l = 1 : l -- if you like lisp, 1:l means (cons 1 l) > > x = f y > y = f x > > main = print x Also, try this: f l = length l : l x = f y y = f x main = print x ;) From benc at hawaga.org.uk Sat Feb 20 04:17:39 2010 From: benc at hawaga.org.uk (Ben Clifford) Date: Sat, 20 Feb 2010 10:17:39 +0000 (GMT) Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266613276.31696.1.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> <4B7EC377.1030105@eecs.northwestern.edu> <1266613276.31696.1.camel@localhost> Message-ID: > > x and y are both infinite lists [1,1,1,1,1,1,1,1,1,1,1,1,1,...] (so > > running the above code prints out a never-ending (until ctrl-c) list of > > 1,1,1,1,...) > > What difference does that make? To the outside observer both programs > hang. the print in the original hangs. print (take 5 x) doesn't hang, despite the fact that x is an infinite list. It prints [1,1,1,1,1] My point basically is that there are some safe uses of cyclic definitions, in languages that although different bear some strong similarity to SwiftScript. It might be that this applies to SwiftScript too or it might be sufficiently different (eg in data structures and in evaluation semantics) that it doesn't apply. -- From mmaiensc at uchicago.edu Wed Feb 24 11:12:05 2010 From: mmaiensc at uchicago.edu (Mark Maienschein-Cline) Date: Wed, 24 Feb 2010 11:12:05 -0600 Subject: [Swift-user] Problems reading file lists in swift Message-ID: I'm having a couple problems calling programs I've written using swift. The first happens for programs where one input is just a text file that is a list of filenames, and the program extracts each filename from the list and opens it for analysis. The programs assume each file is in the same directory as the list, so it gets the directory used to identify the list as well, and adds it to the filename. When I run these on swift and refer to the list by the full path, swift doesn't seems to drop the first "/", so I have to add another one. e.g. "/home/mmaiensc/list" gets turned into "home/mmaiensc/list" and doesn't work, but "//home/mmaiensc/list" gets turned into "/home/mmaiensc/list" and works. Why is this? I'm also having a problem where swift fails, citing "Exit code 5" or "Exit code 10", but thet stdout it gives for the program I called indicates that it finished running correctly (I had it print "done" on the last line). Thanks for any help, Mark From hategan at mcs.anl.gov Wed Feb 24 11:23:35 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Wed, 24 Feb 2010 11:23:35 -0600 Subject: [Swift-user] Problems reading file lists in swift In-Reply-To: References: Message-ID: <1267032215.3652.9.camel@localhost> On Wed, 2010-02-24 at 11:12 -0600, Mark Maienschein-Cline wrote: > I'm having a couple problems calling programs I've written using > swift. > > The first happens for programs where one input is just a text file > that is a list of filenames, and the program extracts each filename > from the list and opens it for analysis. The programs assume each file > is in the same directory as the list, What would happen if you job ran on a remote site where your home directory is named "/users/5/x554332"? > so it gets the directory used to identify the list as well, and adds > it to the filename. When I run these on swift and refer to the list by > the full path, swift doesn't seems to drop the first "/", so I have to > add another one. e.g. "/home/mmaiensc/list" gets turned into > "home/mmaiensc/list" and doesn't work, but "//home/mmaiensc/list" gets > turned into "/home/mmaiensc/list" and works. Why is this? That's a bug. Neither should work. Can you post your swift file? It's hard to figure out exactly what a solution would be without that. In principle, the file list needs to be put in an array and (either individual files in it or the whole array) that needs to be passed to the application such that swift can keep track of files. > > I'm also having a problem where swift fails, citing "Exit code 5" or > "Exit code 10", but thet stdout it gives for the program I called > indicates that it finished running correctly (I had it print "done" on > the last line). Do you have log files for that? Is there any chance of the application both printing "done" and exiting with a non-zero exit code? From wilde at mcs.anl.gov Thu Feb 25 14:46:41 2010 From: wilde at mcs.anl.gov (wilde at mcs.anl.gov) Date: Thu, 25 Feb 2010 14:46:41 -0600 (CST) Subject: [Swift-user] How to set provider-specific options? In-Reply-To: <12952681.907031267130660211.JavaMail.root@zimbra> Message-ID: <14164845.907161267130801693.JavaMail.root@zimbra> I tried to set debug=true as the last line in etc/provider-pbs.properties in my swift dist directory, hoping to get the PBS provider to retain the .submit file in ~/.globus/scripts. That didnt seem to work: I didnt get a .submit file there. Did I do this correctly? - Mike From hategan at mcs.anl.gov Thu Feb 25 14:52:29 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 25 Feb 2010 14:52:29 -0600 Subject: [Swift-user] Re: How to set provider-specific options? In-Reply-To: <14164845.907161267130801693.JavaMail.root@zimbra> References: <14164845.907161267130801693.JavaMail.root@zimbra> Message-ID: <1267131149.28684.4.camel@localhost> On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote: > I tried to set debug=true as the last line in > etc/provider-pbs.properties in my swift dist directory, hoping to get > the PBS provider to retain the .submit file in ~/.globus/scripts. > That didnt seem to work: I didnt get a .submit file there. I don't see any code that would prevent the deletion of the script if debug was set to true in that file. What I would suggest is submitting some long-ish job and then copying the submit scripts from ~/.globus/scripts before they go away. From wilde at mcs.anl.gov Thu Feb 25 14:58:36 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 25 Feb 2010 14:58:36 -0600 (CST) Subject: [Swift-user] Re: How to set provider-specific options? In-Reply-To: <1267131149.28684.4.camel@localhost> Message-ID: <20594772.908021267131516502.JavaMail.root@zimbra> Did I mis-interpret the following code in AbstractExecutor.java? protected void cleanup() { if (!getProperties().isDebugEnabled()) { script.delete(); new File(exitcode).delete(); if (spec.getStdOutput() == null && stdout != null) { new File(stdout).delete(); } if (spec.getStdError() == null && stderr != null) { new File(stderr).delete(); } } } - Mike ----- "Mihael Hategan" wrote: > On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote: > > I tried to set debug=true as the last line in > > etc/provider-pbs.properties in my swift dist directory, hoping to > get > > the PBS provider to retain the .submit file in ~/.globus/scripts. > > That didnt seem to work: I didnt get a .submit file there. > > I don't see any code that would prevent the deletion of the script if > debug was set to true in that file. > > What I would suggest is submitting some long-ish job and then copying > the submit scripts from ~/.globus/scripts before they go away. From hategan at mcs.anl.gov Thu Feb 25 15:06:14 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 25 Feb 2010 15:06:14 -0600 Subject: [Swift-user] Re: How to set provider-specific options? In-Reply-To: <20594772.908021267131516502.JavaMail.root@zimbra> References: <20594772.908021267131516502.JavaMail.root@zimbra> Message-ID: <1267131974.29321.0.camel@localhost> No. But you missed: script = File.createTempFile(getName(), ".submit", scriptdir); script.deleteOnExit(); On Thu, 2010-02-25 at 14:58 -0600, Michael Wilde wrote: > Did I mis-interpret the following code in AbstractExecutor.java? > > protected void cleanup() { > if (!getProperties().isDebugEnabled()) { > script.delete(); > new File(exitcode).delete(); > if (spec.getStdOutput() == null && stdout != null) { > new File(stdout).delete(); > } > if (spec.getStdError() == null && stderr != null) { > new File(stderr).delete(); > } > } > } > > - Mike > > ----- "Mihael Hategan" wrote: > > > On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote: > > > I tried to set debug=true as the last line in > > > etc/provider-pbs.properties in my swift dist directory, hoping to > > get > > > the PBS provider to retain the .submit file in ~/.globus/scripts. > > > That didnt seem to work: I didnt get a .submit file there. > > > > I don't see any code that would prevent the deletion of the script if > > debug was set to true in that file. > > > > What I would suggest is submitting some long-ish job and then copying > > the submit scripts from ~/.globus/scripts before they go away. From wilde at mcs.anl.gov Thu Feb 25 15:59:06 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 25 Feb 2010 15:59:06 -0600 (CST) Subject: [Swift-user] Re: How to set provider-specific options? In-Reply-To: <1267131974.29321.0.camel@localhost> Message-ID: <18355850.911341267135146292.JavaMail.root@zimbra> Do you see any reason not to make the script.deleteOnExit() conditional on the value of the debug= property? - Mike ----- "Mihael Hategan" wrote: > No. But you missed: > script = File.createTempFile(getName(), ".submit", scriptdir); > script.deleteOnExit(); > > On Thu, 2010-02-25 at 14:58 -0600, Michael Wilde wrote: > > Did I mis-interpret the following code in AbstractExecutor.java? > > > > protected void cleanup() { > > if (!getProperties().isDebugEnabled()) { > > script.delete(); > > new File(exitcode).delete(); > > if (spec.getStdOutput() == null && stdout != > null) { > > new File(stdout).delete(); > > } > > if (spec.getStdError() == null && stderr != > null) { > > new File(stderr).delete(); > > } > > } > > } > > > > - Mike > > > > ----- "Mihael Hategan" wrote: > > > > > On Thu, 2010-02-25 at 14:46 -0600, wilde at mcs.anl.gov wrote: > > > > I tried to set debug=true as the last line in > > > > etc/provider-pbs.properties in my swift dist directory, hoping > to > > > get > > > > the PBS provider to retain the .submit file in > ~/.globus/scripts. > > > > That didnt seem to work: I didnt get a .submit file there. > > > > > > I don't see any code that would prevent the deletion of the script > if > > > debug was set to true in that file. > > > > > > What I would suggest is submitting some long-ish job and then > copying > > > the submit scripts from ~/.globus/scripts before they go away. From wilde at mcs.anl.gov Thu Feb 25 16:08:13 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 25 Feb 2010 16:08:13 -0600 (CST) Subject: [Swift-user] How to run coasters over ssh:pbs ? In-Reply-To: <4B86EB4C.2070902@ci.uchicago.edu> Message-ID: <6298113.911831267135693389.JavaMail.root@zimbra> Was: Re: [PADS Support #3457] globus services for accessing PADS -- Hi Wenjun, Are you doing this from communicado? If so, you need to use ssh to get from communicado to PADS. And since when you get there you want to run coasters. you need I think to use the coaster provider and jobmanager = ssh:pbs. I'm cc'ing swift-user and Mihael to discuss the exact sites options needed. We'd want coasters to use ssh to launch one PBS job to launch all the workers. I think there are still some issues on PADS with interpreting #nodes and workerspernode correctly, so please bear with us while we figure this out. I'll try this myself later tonight, but if you discover anything (either working of failing) please post to the User list. Thanks, Mike ----- "Wenjun Wu" wrote: > Hi Mike, > I tried to run loop model through ssh+pbs on PADS. The ssh provider > > seems working but pbs not. > Please check the site file and error message in the below: > > Wenjun > > Wenjun, > > > > I think having GRAM (GRAM5 especially) on PADS would be useful. > > > > But in the meantime, I think you can also use swift's ssh provider > to get to PADS. > > We'll need to try that. > > > > - Mike > > > > > > > > > > /gpfs/pads/oops/workflows > 0.03 > 10000 > > > Caused by: qstat failed (exit code 255) > > at > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:36) > at > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:42) > at > org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:147) > at > org.globus.cog.karajan.workflow.nodes.grid.GridExec.taskFailed(GridExec.java:320) > at > org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.statusChanged(AbstractGridNode.java:276) > at > org.griphyn.vdl.karajan.lib.Execute.statusChanged(Execute.java:104) > at > org.globus.cog.karajan.scheduler.AbstractScheduler.fireJobStatusChangeEvent(AbstractScheduler.java:168) > at > org.globus.cog.karajan.scheduler.LateBindingScheduler.statusChanged(LateBindingScheduler.java:661) > at > org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:424) > at > org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:410) > at > org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:236) > at > org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:224) > at > org.globus.cog.abstraction.impl.common.AbstractDelegatedTaskHandler.failTask(AbstractDelegatedTaskHandler.java:54) > at > org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.processFailed(AbstractJobSubmissionTaskHandler.java:101) > at > org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.processFailed(AbstractExecutor.java:246) > at > org.globus.cog.abstraction.impl.scheduler.common.Job.fail(Job.java:198) > at > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.failAll(AbstractQueuePoller.java:141) > at > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.pollQueue(AbstractQueuePoller.java:172) > at > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run(AbstractQueuePoller.java:80) > ... 1 more > > > > ----- "Ti Leggett" wrote: > > > > > >> There are currently no Globus job managers, but GridFTP is at > >> stor.ci.uchicago.edu. > >> > >> I'll work on getting GRAM up. Do you need WSRF or Pre-WS GRAM? > >> > >> On Wed Feb 24 14:45:57 2010, wwj at ci.uchicago.edu wrote: > >> > >>> Hello, > >>> Could you please tell me whether there is any globus services > for > >>> > >>> PADS such as GridFTP and GRAM? > >>> I'm trying to launch jobs to PADS remotely but can't figure > out > >>> > >> what > >> > >>> are the right URLs for the GridFTP > >>> service and GRAM. > >>> Thanks! > >>> > >>> Wenjun > >>> > > > > From hategan at mcs.anl.gov Thu Feb 25 16:15:08 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 25 Feb 2010 16:15:08 -0600 Subject: [Swift-user] Re: How to set provider-specific options? In-Reply-To: <18355850.911341267135146292.JavaMail.root@zimbra> References: <18355850.911341267135146292.JavaMail.root@zimbra> Message-ID: <1267136108.31901.0.camel@localhost> On Thu, 2010-02-25 at 15:59 -0600, Michael Wilde wrote: > Do you see any reason not to make the script.deleteOnExit() conditional on the value of the debug= property? No. I, too, missed that when the debug thing went in. From hategan at mcs.anl.gov Thu Feb 25 16:17:55 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 25 Feb 2010 16:17:55 -0600 Subject: [Swift-user] Re: How to run coasters over ssh:pbs ? In-Reply-To: <6298113.911831267135693389.JavaMail.root@zimbra> References: <6298113.911831267135693389.JavaMail.root@zimbra> Message-ID: <1267136276.31901.2.camel@localhost> I think what's happening in Wenjun's case is that qsub is not in the path. I suppose this could be tested by running a simple ssh job that does "which qsub". On Thu, 2010-02-25 at 16:08 -0600, Michael Wilde wrote: > Was: Re: [PADS Support #3457] globus services for accessing PADS > > -- > > Hi Wenjun, > > Are you doing this from communicado? If so, you need to use ssh to get > from communicado to PADS. And since when you get there you want to run > coasters. you need I think to use the coaster provider and jobmanager > = ssh:pbs. > > I'm cc'ing swift-user and Mihael to discuss the exact sites options > needed. > > We'd want coasters to use ssh to launch one PBS job to launch all the > workers. > > I think there are still some issues on PADS with interpreting #nodes > and workerspernode correctly, so please bear with us while we figure > this out. > > I'll try this myself later tonight, but if you discover anything > (either working of failing) please post to the User list. > > Thanks, > > Mike > > > ----- "Wenjun Wu" wrote: > > > Hi Mike, > > I tried to run loop model through ssh+pbs on PADS. The ssh provider > > > > seems working but pbs not. > > Please check the site file and error message in the below: > > > > Wenjun > > > Wenjun, > > > > > > I think having GRAM (GRAM5 especially) on PADS would be useful. > > > > > > But in the meantime, I think you can also use swift's ssh provider > > to get to PADS. > > > We'll need to try that. > > > > > > - Mike > > > > > > > > > > > > > > > > > > /gpfs/pads/oops/workflows > > 0.03 > > 10000 > > > > > > Caused by: qstat failed (exit code 255) > > > > at > > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:36) > > at > > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:42) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:147) > > at > > org.globus.cog.karajan.workflow.nodes.grid.GridExec.taskFailed(GridExec.java:320) > > at > > org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.statusChanged(AbstractGridNode.java:276) > > at > > org.griphyn.vdl.karajan.lib.Execute.statusChanged(Execute.java:104) > > at > > org.globus.cog.karajan.scheduler.AbstractScheduler.fireJobStatusChangeEvent(AbstractScheduler.java:168) > > at > > org.globus.cog.karajan.scheduler.LateBindingScheduler.statusChanged(LateBindingScheduler.java:661) > > at > > org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:424) > > at > > org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:410) > > at > > org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:236) > > at > > org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:224) > > at > > org.globus.cog.abstraction.impl.common.AbstractDelegatedTaskHandler.failTask(AbstractDelegatedTaskHandler.java:54) > > at > > org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.processFailed(AbstractJobSubmissionTaskHandler.java:101) > > at > > org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.processFailed(AbstractExecutor.java:246) > > at > > org.globus.cog.abstraction.impl.scheduler.common.Job.fail(Job.java:198) > > at > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.failAll(AbstractQueuePoller.java:141) > > at > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.pollQueue(AbstractQueuePoller.java:172) > > at > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run(AbstractQueuePoller.java:80) > > ... 1 more > > > > > > > ----- "Ti Leggett" wrote: > > > > > > > > >> There are currently no Globus job managers, but GridFTP is at > > >> stor.ci.uchicago.edu. > > >> > > >> I'll work on getting GRAM up. Do you need WSRF or Pre-WS GRAM? > > >> > > >> On Wed Feb 24 14:45:57 2010, wwj at ci.uchicago.edu wrote: > > >> > > >>> Hello, > > >>> Could you please tell me whether there is any globus services > > for > > >>> > > >>> PADS such as GridFTP and GRAM? > > >>> I'm trying to launch jobs to PADS remotely but can't figure > > out > > >>> > > >> what > > >> > > >>> are the right URLs for the GridFTP > > >>> service and GRAM. > > >>> Thanks! > > >>> > > >>> Wenjun > > >>> > > > > > > From iraicu at cs.uchicago.edu Thu Feb 25 17:06:33 2010 From: iraicu at cs.uchicago.edu (Ioan Raicu) Date: Thu, 25 Feb 2010 17:06:33 -0600 Subject: [Swift-user] ACM ScienceCloud2010 deadline extension to 03/15/10 Message-ID: <4B870279.1050406@cs.uchicago.edu> Call for Papers --------------------------------------------------------------------------------------- 1st ACM Workshop on Scientific Cloud Computing (ScienceCloud) 2010 http://dsl.cs.uchicago.edu/ScienceCloud2010/ --------------------------------------------------------------------------------------- June 21st, 2010 Chicago, Illinois, USA Co-located with with ACM High Performance Distributed Computing Conference (HPDC) 2010 Topics of Interest --------------------------------------------------------------------------------------- We invite the submission of original work that is related to the topics below. The papers can be either short (5 pages) position papers, or long (10 pages) research papers. Topics of interest include (in the context of Cloud Computing): * scientific computing applications * performance evaluation * programming models and tools * storage cloud architectures and implementations * compute resource management * high-performance computing * models, frameworks and systems for cloud security Important Dates --------------------------------------------------------------------------------------- * Abstract Due: March 15th, 2010 * Papers Due: March 15th, 2010 * Notification of Acceptance: April 15th, 2010 * Camera Ready Papers Due: May 1st, 2010 * Workshop Date: June 21st, 2010 Committee Members --------------------------------------------------------------------------------------- Workshop Chairs * Pete Beckman, University of Chicago & Argonne National Laboratory * Ian Foster, University of Chicago & Argonne National Laboratory * Ioan Raicu, Northwestern University Steering Committee * Jeff Broughton, Lawrence Berkeley National Lab., USA * Alok Choudhary, Northwestern University, USA * Dennis Gannon, Microsoft Research, USA * Robert Grossman, University of Illinois at Chicago, USA * Kate Keahey, Nimbus, University of Chicago, Argonne National Laboratory, USA * Ed Lazowska, University of Washington, USA * Ignacio Llorente, Open Nebula, Universidad Complutense de Madrid, Spain * David E. Martin, Argonne National Laboratory, Northwestern University, USA * Gabriel Mateescu, Linkoping University, Sweden * David O'Hallaron, Carnegie Mellon University, Intel Labs, USA * Rich Wolski, Eucalyptus, University of California, Santa Barbara, USA * Kathy Yelick, University of California at Berkeley, Lawrence Berkeley National Lab., USA -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From wwj at ci.uchicago.edu Thu Feb 25 17:16:53 2010 From: wwj at ci.uchicago.edu (Wenjun Wu) Date: Thu, 25 Feb 2010 17:16:53 -0600 Subject: [Swift-user] Re: How to run coasters over ssh:pbs ? In-Reply-To: <1267136276.31901.2.camel@localhost> References: <6298113.911831267135693389.JavaMail.root@zimbra> <1267136276.31901.2.camel@localhost> Message-ID: <4B8704E5.4020301@ci.uchicago.edu> Hi Mihael, Please find the attached log file and sites.xml. I run the swift script from sidgrid to pads. And ssh "which qsub" returns the right path. But the coaster didn't work for me. Thanks! Wenjun > I think what's happening in Wenjun's case is that qsub is not in the > path. I suppose this could be tested by running a simple ssh job that > does "which qsub". > > On Thu, 2010-02-25 at 16:08 -0600, Michael Wilde wrote: > >> Was: Re: [PADS Support #3457] globus services for accessing PADS >> >> -- >> >> Hi Wenjun, >> >> Are you doing this from communicado? If so, you need to use ssh to get >> from communicado to PADS. And since when you get there you want to run >> coasters. you need I think to use the coaster provider and jobmanager >> = ssh:pbs. >> >> I'm cc'ing swift-user and Mihael to discuss the exact sites options >> needed. >> >> We'd want coasters to use ssh to launch one PBS job to launch all the >> workers. >> >> I think there are still some issues on PADS with interpreting #nodes >> and workerspernode correctly, so please bear with us while we figure >> this out. >> >> I'll try this myself later tonight, but if you discover anything >> (either working of failing) please post to the User list. >> >> Thanks, >> >> Mike >> >> >> ----- "Wenjun Wu" wrote: >> >> >>> Hi Mike, >>> I tried to run loop model through ssh+pbs on PADS. The ssh provider >>> >>> seems working but pbs not. >>> Please check the site file and error message in the below: >>> >>> Wenjun >>> >>>> Wenjun, >>>> >>>> I think having GRAM (GRAM5 especially) on PADS would be useful. >>>> >>>> But in the meantime, I think you can also use swift's ssh provider >>>> >>> to get to PADS. >>> >>>> We'll need to try that. >>>> >>>> - Mike >>>> >>>> >>>> >>> >>> >>> >>> >>> /gpfs/pads/oops/workflows >>> 0.03 >>> 10000 >>> >>> >>> Caused by: qstat failed (exit code 255) >>> >>> at >>> org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:36) >>> at >>> org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:42) >>> at >>> org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:147) >>> at >>> org.globus.cog.karajan.workflow.nodes.grid.GridExec.taskFailed(GridExec.java:320) >>> at >>> org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.statusChanged(AbstractGridNode.java:276) >>> at >>> org.griphyn.vdl.karajan.lib.Execute.statusChanged(Execute.java:104) >>> at >>> org.globus.cog.karajan.scheduler.AbstractScheduler.fireJobStatusChangeEvent(AbstractScheduler.java:168) >>> at >>> org.globus.cog.karajan.scheduler.LateBindingScheduler.statusChanged(LateBindingScheduler.java:661) >>> at >>> org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:424) >>> at >>> org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:410) >>> at >>> org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:236) >>> at >>> org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:224) >>> at >>> org.globus.cog.abstraction.impl.common.AbstractDelegatedTaskHandler.failTask(AbstractDelegatedTaskHandler.java:54) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.processFailed(AbstractJobSubmissionTaskHandler.java:101) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.processFailed(AbstractExecutor.java:246) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.Job.fail(Job.java:198) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.failAll(AbstractQueuePoller.java:141) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.pollQueue(AbstractQueuePoller.java:172) >>> at >>> org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run(AbstractQueuePoller.java:80) >>> ... 1 more >>> >>> >>> >>>> ----- "Ti Leggett" wrote: >>>> >>>> >>>> >>>>> There are currently no Globus job managers, but GridFTP is at >>>>> stor.ci.uchicago.edu. >>>>> >>>>> I'll work on getting GRAM up. Do you need WSRF or Pre-WS GRAM? >>>>> >>>>> On Wed Feb 24 14:45:57 2010, wwj at ci.uchicago.edu wrote: >>>>> >>>>> >>>>>> Hello, >>>>>> Could you please tell me whether there is any globus services >>>>>> >>> for >>> >>>>>> >>>>>> PADS such as GridFTP and GRAM? >>>>>> I'm trying to launch jobs to PADS remotely but can't figure >>>>>> >>> out >>> >>>>>> >>>>>> >>>>> what >>>>> >>>>> >>>>>> are the right URLs for the GridFTP >>>>>> service and GRAM. >>>>>> Thanks! >>>>>> >>>>>> Wenjun >>>>>> >>>>>> >>>> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: sites.xml Type: text/xml Size: 2314 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: psim.loops-20100225-1712-3zrtb5i5.log URL: From wilde at mcs.anl.gov Thu Feb 25 17:43:36 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 25 Feb 2010 17:43:36 -0600 (CST) Subject: [Swift-user] Re: How to run coasters over ssh:pbs ? In-Reply-To: <1267136276.31901.2.camel@localhost> Message-ID: <28029268.916061267141416696.JavaMail.root@zimbra> But I think in addition Wenjun was running this from communicado, his portal host (and ie no local qsub) so he needs to ssh to PADS and run qsub there (ie, get the coaster provider to do that). - Mike ----- "Mihael Hategan" wrote: > I think what's happening in Wenjun's case is that qsub is not in the > path. I suppose this could be tested by running a simple ssh job that > does "which qsub". > > On Thu, 2010-02-25 at 16:08 -0600, Michael Wilde wrote: > > Was: Re: [PADS Support #3457] globus services for accessing PADS > > > > -- > > > > Hi Wenjun, > > > > Are you doing this from communicado? If so, you need to use ssh to > get > > from communicado to PADS. And since when you get there you want to > run > > coasters. you need I think to use the coaster provider and > jobmanager > > = ssh:pbs. > > > > I'm cc'ing swift-user and Mihael to discuss the exact sites options > > needed. > > > > We'd want coasters to use ssh to launch one PBS job to launch all > the > > workers. > > > > I think there are still some issues on PADS with interpreting > #nodes > > and workerspernode correctly, so please bear with us while we > figure > > this out. > > > > I'll try this myself later tonight, but if you discover anything > > (either working of failing) please post to the User list. > > > > Thanks, > > > > Mike > > > > > > ----- "Wenjun Wu" wrote: > > > > > Hi Mike, > > > I tried to run loop model through ssh+pbs on PADS. The ssh > provider > > > > > > seems working but pbs not. > > > Please check the site file and error message in the below: > > > > > > Wenjun > > > > Wenjun, > > > > > > > > I think having GRAM (GRAM5 especially) on PADS would be useful. > > > > > > > > But in the meantime, I think you can also use swift's ssh > provider > > > to get to PADS. > > > > We'll need to try that. > > > > > > > > - Mike > > > > > > > > > > > > > > > > > > > > /> > > > > > > /gpfs/pads/oops/workflows > > > 0.03 > > > key="initialScore">10000 > > > > > > > > > Caused by: qstat failed (exit code 255) > > > > > > at > > > > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:36) > > > at > > > > org.globus.cog.karajan.workflow.events.FailureNotificationEvent.(FailureNotificationEvent.java:42) > > > at > > > > org.globus.cog.karajan.workflow.nodes.FlowNode.failImmediately(FlowNode.java:147) > > > at > > > > org.globus.cog.karajan.workflow.nodes.grid.GridExec.taskFailed(GridExec.java:320) > > > at > > > > org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.statusChanged(AbstractGridNode.java:276) > > > at > > > > org.griphyn.vdl.karajan.lib.Execute.statusChanged(Execute.java:104) > > > at > > > > org.globus.cog.karajan.scheduler.AbstractScheduler.fireJobStatusChangeEvent(AbstractScheduler.java:168) > > > at > > > > org.globus.cog.karajan.scheduler.LateBindingScheduler.statusChanged(LateBindingScheduler.java:661) > > > at > > > > org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:424) > > > at > > > > org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:410) > > > at > > > > org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:236) > > > at > > > > org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:224) > > > at > > > > org.globus.cog.abstraction.impl.common.AbstractDelegatedTaskHandler.failTask(AbstractDelegatedTaskHandler.java:54) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.processFailed(AbstractJobSubmissionTaskHandler.java:101) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.processFailed(AbstractExecutor.java:246) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.Job.fail(Job.java:198) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.failAll(AbstractQueuePoller.java:141) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.pollQueue(AbstractQueuePoller.java:172) > > > at > > > > org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run(AbstractQueuePoller.java:80) > > > ... 1 more > > > > > > > > > > ----- "Ti Leggett" wrote: > > > > > > > > > > > >> There are currently no Globus job managers, but GridFTP is at > > > >> stor.ci.uchicago.edu. > > > >> > > > >> I'll work on getting GRAM up. Do you need WSRF or Pre-WS GRAM? > > > >> > > > >> On Wed Feb 24 14:45:57 2010, wwj at ci.uchicago.edu wrote: > > > >> > > > >>> Hello, > > > >>> Could you please tell me whether there is any globus > services > > > for > > > >>> > > > >>> PADS such as GridFTP and GRAM? > > > >>> I'm trying to launch jobs to PADS remotely but can't > figure > > > out > > > >>> > > > >> what > > > >> > > > >>> are the right URLs for the GridFTP > > > >>> service and GRAM. > > > >>> Thanks! > > > >>> > > > >>> Wenjun > > > >>> > > > > > > > > From mmaiensc at uchicago.edu Fri Feb 26 14:31:30 2010 From: mmaiensc at uchicago.edu (Mark Maienschein-Cline) Date: Fri, 26 Feb 2010 14:31:30 -0600 Subject: [Swift-user] Directory mappers Message-ID: <4A2A6130-C88F-43FC-AB00-420A4DF8BFC8@uchicago.edu> Hi, I have a large number of data directories, each with a specific file I would need to run analysis on. For example, I have directories expt1/, expt2/, expt3/, ..., and the files I need to map are in "expt1/output/data.txt" (and the same for the other experiments). The file name is always the same, but the directory changes. Is there a mapper that can get all of the "data.txt" files for me? Thanks, Mark From aespinosa at cs.uchicago.edu Fri Feb 26 15:51:25 2010 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 26 Feb 2010 15:51:25 -0600 Subject: [Swift-user] Directory mappers In-Reply-To: <4A2A6130-C88F-43FC-AB00-420A4DF8BFC8@uchicago.edu> References: <4A2A6130-C88F-43FC-AB00-420A4DF8BFC8@uchicago.edu> Message-ID: <20100226215125.GA3922@origin> Hi Mark, For most of my complex input output data specifications, I always use the ext_mapper since I can always make a script to specify what kind of data i want. a typical example script would be #!/bin/bash n=35 for i in `seq 0 $n`; do [$i] expt$i/data.txt done -Allan On Fri, Feb 26, 2010 at 02:31:30PM -0600, Mark Maienschein-Cline wrote: > Hi, > I have a large number of data directories, each with a specific file I would > need to run analysis on. For example, I have directories expt1/, expt2/, > expt3/, ..., and the files I need to map are in "expt1/output/data.txt" (and > the same for the other experiments). The file name is always the same, but the > directory changes. Is there a mapper that can get all of the "data.txt" files > for me? > > > Thanks, > Mark_______________________________________________ From wilde at mcs.anl.gov Sat Feb 27 13:02:06 2010 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 27 Feb 2010 13:02:06 -0600 (CST) Subject: [Swift-user] Directory mappers In-Reply-To: <20100226215125.GA3922@origin> Message-ID: <14428723.962211267297326766.JavaMail.root@zimbra> Right, I agree, ext_mapper is what you want here, Mark. Documented in the user guide at: http://www.ci.uchicago.edu/swift/guides/userguide.php#mapper.ext_mapper You can pass parameters such as the location of the higher parent directory. When using an ext mapper as an output mapper for an array, you need (at the moment) to know in advance how many members you are expecting. We're discussing ways to remove that limitation. - Mike ----- "Allan Espinosa" wrote: > Hi Mark, > > For most of my complex input output data specifications, I always use > the > ext_mapper since I can always make a script to specify what kind of > data i want. > a typical example script would be > > #!/bin/bash > > n=35 > for i in `seq 0 $n`; do > [$i] expt$i/data.txt > done > > -Allan > > > On Fri, Feb 26, 2010 at 02:31:30PM -0600, Mark Maienschein-Cline > wrote: > > Hi, > > I have a large number of data directories, each with a specific file > I would > > need to run analysis on. For example, I have directories expt1/, > expt2/, > > expt3/, ..., and the files I need to map are in > "expt1/output/data.txt" (and > > the same for the other experiments). The file name is always the > same, but the > > directory changes. Is there a mapper that can get all of the > "data.txt" files > > for me? > > > > > > Thanks, > > Mark_______________________________________________ > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From zhaozhang at uchicago.edu Sun Feb 28 00:08:12 2010 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Sun, 28 Feb 2010 00:08:12 -0600 Subject: [Swift-user] swift returns error on pads Message-ID: <4B8A084C.1090903@uchicago.edu> Hi, I have a following error message from swift when I ran it on pads with coaster. The swift I am running is at [zzhang at login2 final]$ which swift /home/wilde/bigdata/swift/bin/swift I am confused about the configuration settings in the sites.xml file. Could anybody point the error in the settings? thanks. Best zhao PS: I am pasting my pbs.xml and stdout from swift here. [zzhang at login2 final]$ cat pbs.xml 00:00:10 1800 8 10000 .63 /home/zzhang/swiftwork [zzhang at login2 final]$ time swift -tc.file ./tc sites.file ./pbs.xml movie.swift Swift svn swift-r3255 (swift modified locally) cog-r2723 RunID: 20100228-0003-xeza2uh4 Progress: Progress:Progress: uninitialized:3 uninitialized:3 Progress: Selecting site:936 Stage in:54 Submitting:1 Submitted:9 Progress: Selecting site:936 Stage in:21 Submitted:43 Worker task failed: Error submitting block task org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (qsub reported an exit code of 188). qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (qsub reported an exit code of 188). qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) ... 3 more Progress: Selecting site:936 Submitted:63 Active:1 Failed to transfer wrapper log from movie-20100228-0003-xeza2uh4/info/v on pbs Execution failed: Exception in transform: Arguments: [training_set_1000/mv_0001283.txt] Host: pbs Directory: movie-20100228-0003-xeza2uh4/jobs/v/transform-vt0goeoj stderr.txt: stdout.txt: ---- Caused by: Task failed: Error submitting block task org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (qsub reported an exit code of 188). qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (qsub reported an exit code of 188). qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) ... 3 more Cleaning up... Shutting down service at https://192.5.86.6:50002 Got channel MetaChannel: 2090567670 -> null +Canceling job 5536.svc.pads.ci.uchicago.edu Canceling job 5537.svc.pads.ci.uchicago.edu Canceling job 5538.svc.pads.ci.uchicago.edu Failed to shut down block org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:177) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44) at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:487) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:477) at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:70) at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:211) at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:348) at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.sendTaggedData(AbstractPipedChannel.java:58) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.sendTaggedData(AbstractKarajanChannel.java:126) at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:121) at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:171) at org.globus.cog.karajan.workflow.service.commands.Command.executeAsync(Command.java:162) at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:425) Done real 0m14.073s user 0m11.112s sys 0m1.577s From aespinosa at cs.uchicago.edu Sun Feb 28 00:38:05 2010 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sun, 28 Feb 2010 00:38:05 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <4B8A084C.1090903@uchicago.edu> References: <4B8A084C.1090903@uchicago.edu> Message-ID: <20100228063805.GA9126@origin> I think it should be workersPerNode On Sun, Feb 28, 2010 at 12:08:12AM -0600, Zhao Zhang wrote: > Hi, > > I have a following error message from swift when I ran it on pads > with coaster. The swift > I am running is at > [zzhang at login2 final]$ which swift > /home/wilde/bigdata/swift/bin/swift > > I am confused about the configuration settings in the sites.xml > file. Could anybody point the > error in the settings? thanks. > > Best > zhao > > PS: I am pasting my pbs.xml and stdout from swift here. > > [zzhang at login2 final]$ cat pbs.xml > > > > > 00:00:10 > 1800 > > > 8 > > 10000 > .63 > > > /home/zzhang/swiftwork > > > > > > [zzhang at login2 final]$ time swift -tc.file ./tc sites.file ./pbs.xml > movie.swift > Swift svn swift-r3255 (swift modified locally) cog-r2723 > > RunID: 20100228-0003-xeza2uh4 > Progress: > Progress:Progress: uninitialized:3 uninitialized:3 > > Progress: Selecting site:936 Stage in:54 Submitting:1 Submitted:9 > Progress: Selecting site:936 Stage in:21 Submitted:43 > Worker task failed: Error submitting block task > org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > Cannot submit job: Could not submit job (qsub reported an exit code > of 188). > qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > > at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) > at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) > at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) > at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) > Caused by: > org.globus.cog.abstraction.impl.scheduler.common.ProcessException: > Could not submit job (qsub reported an exit code of 188). > qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > > at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) > at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) > ... 3 more > Progress: Selecting site:936 Submitted:63 Active:1 > Failed to transfer wrapper log from > movie-20100228-0003-xeza2uh4/info/v on pbs > Execution failed: > Exception in transform: > Arguments: [training_set_1000/mv_0001283.txt] > Host: pbs > Directory: movie-20100228-0003-xeza2uh4/jobs/v/transform-vt0goeoj > stderr.txt: > > stdout.txt: > > ---- > > Caused by: > Task failed: Error submitting block task > org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > Cannot submit job: Could not submit job (qsub reported an exit code > of 188). > qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > > at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) > at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) > at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) > at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) > Caused by: > org.globus.cog.abstraction.impl.scheduler.common.ProcessException: > Could not submit job (qsub reported an exit code of 188). > qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > > at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) > at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) > ... 3 more > > Cleaning up... > Shutting down service at https://192.5.86.6:50002 > Got channel MetaChannel: 2090567670 -> null > +Canceling job 5536.svc.pads.ci.uchicago.edu > Canceling job 5537.svc.pads.ci.uchicago.edu > Canceling job 5538.svc.pads.ci.uchicago.edu > Failed to shut down block > org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > Can only cancel an active task > at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:177) > at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) > at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70) > at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96) > at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85) > at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44) > at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271) > at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252) > at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:487) > at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:477) > at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:70) > at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:211) > at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28) > at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84) > at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:348) > at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.sendTaggedData(AbstractPipedChannel.java:58) > at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.sendTaggedData(AbstractKarajanChannel.java:126) > at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:121) > at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:171) > at org.globus.cog.karajan.workflow.service.commands.Command.executeAsync(Command.java:162) > at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:425) From zhaozhang at uchicago.edu Sun Feb 28 00:45:41 2010 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Sun, 28 Feb 2010 00:45:41 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <20100228063805.GA9126@origin> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> Message-ID: <4B8A1115.7040803@uchicago.edu> Thanks, Allan. But I still got the same error. zhao Allan Espinosa wrote: > I think it should be workersPerNode > > On Sun, Feb 28, 2010 at 12:08:12AM -0600, Zhao Zhang wrote: > >> Hi, >> >> I have a following error message from swift when I ran it on pads >> with coaster. The swift >> I am running is at >> [zzhang at login2 final]$ which swift >> /home/wilde/bigdata/swift/bin/swift >> >> I am confused about the configuration settings in the sites.xml >> file. Could anybody point the >> error in the settings? thanks. >> >> Best >> zhao >> >> PS: I am pasting my pbs.xml and stdout from swift here. >> >> [zzhang at login2 final]$ cat pbs.xml >> >> >> >> >> 00:00:10 >> 1800 >> >> >> 8 >> >> 10000 >> .63 >> >> >> /home/zzhang/swiftwork >> >> >> >> >> >> [zzhang at login2 final]$ time swift -tc.file ./tc sites.file ./pbs.xml >> movie.swift >> Swift svn swift-r3255 (swift modified locally) cog-r2723 >> >> RunID: 20100228-0003-xeza2uh4 >> Progress: >> Progress:Progress: uninitialized:3 uninitialized:3 >> >> Progress: Selecting site:936 Stage in:54 Submitting:1 Submitted:9 >> Progress: Selecting site:936 Stage in:21 Submitted:43 >> Worker task failed: Error submitting block task >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >> Cannot submit job: Could not submit job (qsub reported an exit code >> of 188). >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >> >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) >> Caused by: >> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: >> Could not submit job (qsub reported an exit code of 188). >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >> >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) >> ... 3 more >> Progress: Selecting site:936 Submitted:63 Active:1 >> Failed to transfer wrapper log from >> movie-20100228-0003-xeza2uh4/info/v on pbs >> Execution failed: >> Exception in transform: >> Arguments: [training_set_1000/mv_0001283.txt] >> Host: pbs >> Directory: movie-20100228-0003-xeza2uh4/jobs/v/transform-vt0goeoj >> stderr.txt: >> >> stdout.txt: >> >> ---- >> >> Caused by: >> Task failed: Error submitting block task >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >> Cannot submit job: Could not submit job (qsub reported an exit code >> of 188). >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >> >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) >> Caused by: >> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: >> Could not submit job (qsub reported an exit code of 188). >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >> >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) >> ... 3 more >> >> Cleaning up... >> Shutting down service at https://192.5.86.6:50002 >> Got channel MetaChannel: 2090567670 -> null >> +Canceling job 5536.svc.pads.ci.uchicago.edu >> Canceling job 5537.svc.pads.ci.uchicago.edu >> Canceling job 5538.svc.pads.ci.uchicago.edu >> Failed to shut down block >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >> Can only cancel an active task >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:177) >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70) >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96) >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85) >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44) >> at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271) >> at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252) >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:487) >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:477) >> at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:70) >> at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:211) >> at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28) >> at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84) >> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:348) >> at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.sendTaggedData(AbstractPipedChannel.java:58) >> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.sendTaggedData(AbstractKarajanChannel.java:126) >> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:121) >> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:171) >> at org.globus.cog.karajan.workflow.service.commands.Command.executeAsync(Command.java:162) >> at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:425) >> > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > From hategan at mcs.anl.gov Sun Feb 28 01:10:05 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 28 Feb 2010 01:10:05 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <4B8A1115.7040803@uchicago.edu> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> <4B8A1115.7040803@uchicago.edu> Message-ID: <1267341005.3531.0.camel@localhost> The poor thing is allocating more nodes than are available. Specify maxnodes. On Sun, 2010-02-28 at 00:45 -0600, Zhao Zhang wrote: > Thanks, Allan. But I still got the same error. > > zhao > > Allan Espinosa wrote: > > I think it should be workersPerNode > > > > On Sun, Feb 28, 2010 at 12:08:12AM -0600, Zhao Zhang wrote: > > > >> Hi, > >> > >> I have a following error message from swift when I ran it on pads > >> with coaster. The swift > >> I am running is at > >> [zzhang at login2 final]$ which swift > >> /home/wilde/bigdata/swift/bin/swift > >> > >> I am confused about the configuration settings in the sites.xml > >> file. Could anybody point the > >> error in the settings? thanks. > >> > >> Best > >> zhao > >> > >> PS: I am pasting my pbs.xml and stdout from swift here. > >> > >> [zzhang at login2 final]$ cat pbs.xml > >> > >> > >> > >> > >> 00:00:10 > >> 1800 > >> > >> > >> 8 > >> > >> 10000 > >> .63 > >> > >> > >> /home/zzhang/swiftwork > >> > >> > >> > >> > >> > >> [zzhang at login2 final]$ time swift -tc.file ./tc sites.file ./pbs.xml > >> movie.swift > >> Swift svn swift-r3255 (swift modified locally) cog-r2723 > >> > >> RunID: 20100228-0003-xeza2uh4 > >> Progress: > >> Progress:Progress: uninitialized:3 uninitialized:3 > >> > >> Progress: Selecting site:936 Stage in:54 Submitting:1 Submitted:9 > >> Progress: Selecting site:936 Stage in:21 Submitted:43 > >> Worker task failed: Error submitting block task > >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > >> Cannot submit job: Could not submit job (qsub reported an exit code > >> of 188). > >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > >> > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) > >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) > >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) > >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) > >> Caused by: > >> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: > >> Could not submit job (qsub reported an exit code of 188). > >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > >> > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) > >> ... 3 more > >> Progress: Selecting site:936 Submitted:63 Active:1 > >> Failed to transfer wrapper log from > >> movie-20100228-0003-xeza2uh4/info/v on pbs > >> Execution failed: > >> Exception in transform: > >> Arguments: [training_set_1000/mv_0001283.txt] > >> Host: pbs > >> Directory: movie-20100228-0003-xeza2uh4/jobs/v/transform-vt0goeoj > >> stderr.txt: > >> > >> stdout.txt: > >> > >> ---- > >> > >> Caused by: > >> Task failed: Error submitting block task > >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > >> Cannot submit job: Could not submit job (qsub reported an exit code > >> of 188). > >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > >> > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) > >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) > >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) > >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) > >> Caused by: > >> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: > >> Could not submit job (qsub reported an exit code of 188). > >> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes > >> > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) > >> ... 3 more > >> > >> Cleaning up... > >> Shutting down service at https://192.5.86.6:50002 > >> Got channel MetaChannel: 2090567670 -> null > >> +Canceling job 5536.svc.pads.ci.uchicago.edu > >> Canceling job 5537.svc.pads.ci.uchicago.edu > >> Canceling job 5538.svc.pads.ci.uchicago.edu > >> Failed to shut down block > >> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: > >> Can only cancel an active task > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:177) > >> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) > >> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70) > >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96) > >> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85) > >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44) > >> at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271) > >> at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252) > >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:487) > >> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:477) > >> at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:70) > >> at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:211) > >> at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28) > >> at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84) > >> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:348) > >> at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.sendTaggedData(AbstractPipedChannel.java:58) > >> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.sendTaggedData(AbstractKarajanChannel.java:126) > >> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:121) > >> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:171) > >> at org.globus.cog.karajan.workflow.service.commands.Command.executeAsync(Command.java:162) > >> at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:425) > >> > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From zhaozhang at uchicago.edu Sun Feb 28 10:38:29 2010 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Sun, 28 Feb 2010 10:38:29 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <1267341005.3531.0.camel@localhost> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> <4B8A1115.7040803@uchicago.edu> <1267341005.3531.0.camel@localhost> Message-ID: <4B8A9C05.4040408@uchicago.edu> Hi, Mihael It worked. But it only worked if I set the "maxnodes" to "2", does that mean "2" is the only number of compute nodes that I could use on pads? Thanks. Best zhao [zzhang at login2 final]$ cat pbs.xml 00:00:10 1800 8 2 1000 .63 /home/zzhang/swiftwork Mihael Hategan wrote: > The poor thing is allocating more nodes than are available. Specify > maxnodes. > > On Sun, 2010-02-28 at 00:45 -0600, Zhao Zhang wrote: > >> Thanks, Allan. But I still got the same error. >> >> zhao >> >> Allan Espinosa wrote: >> >>> I think it should be workersPerNode >>> >>> On Sun, Feb 28, 2010 at 12:08:12AM -0600, Zhao Zhang wrote: >>> >>> >>>> Hi, >>>> >>>> I have a following error message from swift when I ran it on pads >>>> with coaster. The swift >>>> I am running is at >>>> [zzhang at login2 final]$ which swift >>>> /home/wilde/bigdata/swift/bin/swift >>>> >>>> I am confused about the configuration settings in the sites.xml >>>> file. Could anybody point the >>>> error in the settings? thanks. >>>> >>>> Best >>>> zhao >>>> >>>> PS: I am pasting my pbs.xml and stdout from swift here. >>>> >>>> [zzhang at login2 final]$ cat pbs.xml >>>> >>>> >>>> >>>> >>>> 00:00:10 >>>> 1800 >>>> >>>> >>>> 8 >>>> >>>> 10000 >>>> .63 >>>> >>>> >>>> /home/zzhang/swiftwork >>>> >>>> >>>> >>>> >>>> >>>> [zzhang at login2 final]$ time swift -tc.file ./tc sites.file ./pbs.xml >>>> movie.swift >>>> Swift svn swift-r3255 (swift modified locally) cog-r2723 >>>> >>>> RunID: 20100228-0003-xeza2uh4 >>>> Progress: >>>> Progress:Progress: uninitialized:3 uninitialized:3 >>>> >>>> Progress: Selecting site:936 Stage in:54 Submitting:1 Submitted:9 >>>> Progress: Selecting site:936 Stage in:21 Submitted:43 >>>> Worker task failed: Error submitting block task >>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >>>> Cannot submit job: Could not submit job (qsub reported an exit code >>>> of 188). >>>> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >>>> >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) >>>> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) >>>> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) >>>> Caused by: >>>> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: >>>> Could not submit job (qsub reported an exit code of 188). >>>> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >>>> >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) >>>> ... 3 more >>>> Progress: Selecting site:936 Submitted:63 Active:1 >>>> Failed to transfer wrapper log from >>>> movie-20100228-0003-xeza2uh4/info/v on pbs >>>> Execution failed: >>>> Exception in transform: >>>> Arguments: [training_set_1000/mv_0001283.txt] >>>> Host: pbs >>>> Directory: movie-20100228-0003-xeza2uh4/jobs/v/transform-vt0goeoj >>>> stderr.txt: >>>> >>>> stdout.txt: >>>> >>>> ---- >>>> >>>> Caused by: >>>> Task failed: Error submitting block task >>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >>>> Cannot submit job: Could not submit job (qsub reported an exit code >>>> of 188). >>>> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >>>> >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) >>>> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:46) >>>> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:43) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:66) >>>> Caused by: >>>> org.globus.cog.abstraction.impl.scheduler.common.ProcessException: >>>> Could not submit job (qsub reported an exit code of 188). >>>> qsub: Job exceeds queue resource limits MSG=cannot locate feasible nodes >>>> >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:100) >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53) >>>> ... 3 more >>>> >>>> Cleaning up... >>>> Shutting down service at https://192.5.86.6:50002 >>>> Got channel MetaChannel: 2090567670 -> null >>>> +Canceling job 5536.svc.pads.ci.uchicago.edu >>>> Canceling job 5537.svc.pads.ci.uchicago.edu >>>> Canceling job 5538.svc.pads.ci.uchicago.edu >>>> Failed to shut down block >>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: >>>> Can only cancel an active task >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:177) >>>> at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) >>>> at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:70) >>>> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:96) >>>> at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:85) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:44) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:271) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:252) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:487) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:477) >>>> at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:70) >>>> at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:211) >>>> at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:28) >>>> at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:84) >>>> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:348) >>>> at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.sendTaggedData(AbstractPipedChannel.java:58) >>>> at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.sendTaggedData(AbstractKarajanChannel.java:126) >>>> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:121) >>>> at org.globus.cog.karajan.workflow.service.commands.Command.send(Command.java:171) >>>> at org.globus.cog.karajan.workflow.service.commands.Command.executeAsync(Command.java:162) >>>> at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:425) >>>> >>>> >>> _______________________________________________ >>> Swift-user mailing list >>> Swift-user at ci.uchicago.edu >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >>> >>> >>> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > > > From hategan at mcs.anl.gov Sun Feb 28 12:38:11 2010 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 28 Feb 2010 12:38:11 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <4B8A9C05.4040408@uchicago.edu> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> <4B8A1115.7040803@uchicago.edu> <1267341005.3531.0.camel@localhost> <4B8A9C05.4040408@uchicago.edu> Message-ID: <1267382291.6631.2.camel@localhost> On Sun, 2010-02-28 at 10:38 -0600, Zhao Zhang wrote: > Hi, Mihael > > It worked. But it only worked if I set the "maxnodes" to "2", does that > mean "2" is the only number of > compute nodes that I could use on pads? I don't know what the various limits are on PADS, and I don't know how to find out short of sending mail to support (qstat -Q doesn't say much). But 2 nodes looks suspiciously low. From aespinosa at cs.uchicago.edu Sun Feb 28 14:06:34 2010 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Sun, 28 Feb 2010 14:06:34 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <1267382291.6631.2.camel@localhost> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> <4B8A1115.7040803@uchicago.edu> <1267341005.3531.0.camel@localhost> <4B8A9C05.4040408@uchicago.edu> <1267382291.6631.2.camel@localhost> Message-ID: <20100228200634.GA21363@origin> Also, Zhao you forgot to specify the project account in the pbs.xml -allan On Sun, Feb 28, 2010 at 12:38:11PM -0600, Mihael Hategan wrote: > On Sun, 2010-02-28 at 10:38 -0600, Zhao Zhang wrote: > > Hi, Mihael > > > > It worked. But it only worked if I set the "maxnodes" to "2", does that > > mean "2" is the only number of > > compute nodes that I could use on pads? > > I don't know what the various limits are on PADS, and I don't know how > to find out short of sending mail to support (qstat -Q doesn't say > much). But 2 nodes looks suspiciously low. > > From zhaozhang at uchicago.edu Sun Feb 28 14:13:52 2010 From: zhaozhang at uchicago.edu (Zhao Zhang) Date: Sun, 28 Feb 2010 14:13:52 -0600 Subject: [Swift-user] swift returns error on pads In-Reply-To: <20100228200634.GA21363@origin> References: <4B8A084C.1090903@uchicago.edu> <20100228063805.GA9126@origin> <4B8A1115.7040803@uchicago.edu> <1267341005.3531.0.camel@localhost> <4B8A9C05.4040408@uchicago.edu> <1267382291.6631.2.camel@localhost> <20100228200634.GA21363@origin> Message-ID: <4B8ACE80.1080106@uchicago.edu> Hi, Allan What is our project name for bigdata class? zhao Allan Espinosa wrote: > Also, Zhao you forgot to specify the project account in the pbs.xml > > -allan > On Sun, Feb 28, 2010 at 12:38:11PM -0600, Mihael Hategan wrote: > >> On Sun, 2010-02-28 at 10:38 -0600, Zhao Zhang wrote: >> >>> Hi, Mihael >>> >>> It worked. But it only worked if I set the "maxnodes" to "2", does that >>> mean "2" is the only number of >>> compute nodes that I could use on pads? >>> >> I don't know what the various limits are on PADS, and I don't know how >> to find out short of sending mail to support (qstat -Q doesn't say >> much). But 2 nodes looks suspiciously low. >> >> >> > > From tiberiu.stefpraun at gmail.com Mon Feb 15 10:21:52 2010 From: tiberiu.stefpraun at gmail.com (Tiberiu Stef-Praun) Date: Mon, 15 Feb 2010 16:21:52 -0000 Subject: [Swift-user] Fwd: Delivery Status Notification (Failure) In-Reply-To: <0015175cda8c95db2d047fa5f916@google.com> References: <0015175cda8c95db2d047fa5f916@google.com> Message-ID: Hello Developers/Users I am getting the following exception Karajan exception: Ambiguous element: property. Possible choices: sys:property cdm:property It probably comes from using the concurrent mapper: type statePath{ file iterState[]; } statePath statePathFiles; statePathFiles.iterState[0]=initialize(); I am using swift from SVN Any suggestions ? Thank you Tibi -------------- next part -------------- An HTML attachment was scrubbed... URL: From iraicu at eecs.northwestern.edu Thu Feb 18 17:10:35 2010 From: iraicu at eecs.northwestern.edu (Ioan Raicu) Date: Thu, 18 Feb 2010 23:10:35 -0000 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: <1266532663.17633.0.camel@localhost> References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> Message-ID: <4B7DC3DF.4070107@eecs.northwestern.edu> But I don't get it, how can the variable b not have a value, if it is consumed by cat() in a prior statement? I thought Swift followed a write-once read many paradigm, which was enforced by the single assignment paradigm. Perhaps it doesn't, and it was just my naive interpretation. I guess there is a distinction between files and variables? The single assignment only relates to variables, and not files? I am a bit confused I think, but I am sure it will all become clear, with a bit more explanation from you about how the single assignment fits into SwiftScript. Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Mihael Hategan wrote: > There is no existing value of b. > > Here's the full example, which you can try out: > > app (file y) cat(file x) { > cat @x stdout=@y; > } > > type file; > > file a; > file b; > > a = cat(b); > b = cat(a); > > On Thu, 2010-02-18 at 16:17 -0600, Ioan Raicu wrote: > >> But wouldn't the single assignment part prevent your example from >> dead-locking? In other words, the compiler should throw an error on >> line "b = cat(a)" as its trying to modify an existing value b. >> >> Mihael Hategan wrote: >> >>> On Thu, 2010-02-18 at 17:08 +0000, Ben Clifford wrote: >>> >>> >>>>> I'm unsure. I have a feeling that static dealdock detection is >>>>> undecidable in swift. I need to think some more about that. >>>>> >>>>> >>>> I think doing it statically (i.e. at compile time) is equivalent to >>>> executing the code (including external apps) - I think there's probably a >>>> straightforward example where something depends on an integer read from an >>>> externally executed app which then influences an if statement that decides >>>> which of two different arrays is being written to; or something like that. >>>> >>>> >>> Right. So termination is undecidable by virtue of it being a turing >>> complete language. What I'm unsure is whether deadlocks follow as a >>> consequence of that. They are a different beast. >>> >>> We can probably start with simpler things. The following deadlocks with >>> the current swift: >>> a = cat(b); >>> b = cat(a); >>> >>> So it seems that loops in the graph will create a deadlock and detecting >>> loops requires knowing the graph. We can probably detect the static >>> parts and we may be able to handle alternatives by issuing warnings of >>> the "branch 1 in that 'if' combined with branch 2 in that other 'if' >>> leads to a deadlock" sort. >>> >>> Arrays may be a bit tougher. >>> >>> >>> _______________________________________________ >>> Swift-devel mailing list >>> Swift-devel at ci.uchicago.edu >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel >>> >>> >>> >> -- >> ================================================================= >> Ioan Raicu, Ph.D. >> NSF/CRA Computing Innovation Fellow >> ================================================================= >> Center for Ultra-scale Computing and Information Security (CUCIS) >> Department of Electrical Engineering and Computer Science >> Northwestern University >> 2145 Sheridan Rd, Tech M384 >> Evanston, IL 60208-3118 >> ================================================================= >> Cel: 1-847-722-0876 >> Tel: 1-847-491-8163 >> Email: iraicu at eecs.northwestern.edu >> Web: http://www.eecs.northwestern.edu/~iraicu/ >> https://wiki.cucis.eecs.northwestern.edu/ >> ================================================================= >> ================================================================= >> >> From iraicu at eecs.northwestern.edu Fri Feb 19 10:58:48 2010 From: iraicu at eecs.northwestern.edu (Ioan Raicu) Date: Fri, 19 Feb 2010 16:58:48 -0000 Subject: [Swift-user] Re: [Swift-devel] Problem with iterate In-Reply-To: References: <1266445028.16024.13.camel@localhost> <1266512689.6317.30.camel@localhost> <1266530849.15179.107.camel@localhost> <4B7DBC72.6020403@cs.uchicago.edu> <1266532663.17633.0.camel@localhost> <4B7DC3DF.4070107@eecs.northwestern.edu> <4B7EC0F8.7060005@cs.uchicago.edu> Message-ID: <4B7EC377.1030105@eecs.northwestern.edu> But lets bring this back to a more real example. A user wanting to express some computations that have some dependencies, would write out their computations in some order, expecting their order to be preserved because of the dependencies. If you only support single assignment on variables (e.g. the data), then an example like the one below could never deadlock because the single assignment would be violated on the 2nd statement. Perhaps things are more complicated if you support multiple assignments per variables, but that is not the case for Swift, right? I am trying to understand if this deadlock is happening in Swift due to some particular implementation detail in Swift (or underlying pieces), or is it a fundamental flaw in the DAG based approach with single assignment variables? Or is it due to something completely different? Thanks, Ioan -- ================================================================= Ioan Raicu, Ph.D. NSF/CRA Computing Innovation Fellow ================================================================= Center for Ultra-scale Computing and Information Security (CUCIS) Department of Electrical Engineering and Computer Science Northwestern University 2145 Sheridan Rd, Tech M384 Evanston, IL 60208-3118 ================================================================= Cel: 1-847-722-0876 Tel: 1-847-491-8163 Email: iraicu at eecs.northwestern.edu Web: http://www.eecs.northwestern.edu/~iraicu/ https://wiki.cucis.eecs.northwestern.edu/ ================================================================= ================================================================= Ben Clifford wrote: >> Mathematically, the two different sequences evaluate to different values: >> >> x = y + 1 >> y = x + 1 >> >> assuming y = 0, x = 1 >> > > I mean in simultaneous equations (linear algebra) - in other words, "find > (through whatever means you care to use) a value of x and y such that the > above two equations are both satisfied" - there is no value of x and y > that satisfies that. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: