From iraicu at cs.iit.edu Wed Mar 2 10:47:17 2011 From: iraicu at cs.iit.edu (Ioan Raicu) Date: Wed, 02 Mar 2011 10:47:17 -0600 Subject: [Swift-user] Call for Extended Abstracts: Cloud Computing and its Applications (CCA) 2011 Message-ID: <4D6E7495.5000105@cs.iit.edu> --------------------------------------------------------------------------------- *** Call for Extended Abstracts *** The 3rd Workshop on Cloud Computing and its Applications (CCA) 2011 In conjunction with GlobusWorld 2011, April 12th-13th, Argonne, Illinois http://www.cca11.org/ --------------------------------------------------------------------------------- Dramatic growth in data and equally rapid decline in the cost of highly integrated clusters has spurred the emergence of the data center as the platform of choice for a growing class of data-intensive applications. To encourage conversations between those developing applications, algorithms, software, and hardware for such "cloud" platforms, we are convening the third workshop on Cloud Computing and Its Applications (CCA11). CCA11 will provide reception and poster session on April 12th (Call for Extended Abstract - due March 15th), as well as a full day of distinguished invited talks on April 13th on cloud computing, data intensive scalable computing, and related topics. CCA11 will be held at Argonne National Laboratory (Bldg. 240 Conference Center) in Argonne Illinois, just 25 miles west of Chicago, Illinois. TENTATIVE PROGRAM --------------------------------------------------------------------------------- The tentative program for CCA11 is (the latest updates can be found at http://cca11.org/agenda/): Tuesday, April 12 5:00pm - 7:00pm Reception and Posters Wednesday, April 13 8:15a - 9:00a Continental Breakfast 9:00a - 10:00a Keynote 10:30a - 12:00p Session 1 - Data Center and Cloud Networking -Data Center Networks -Steven Carter, Cisco Virtual Networks 1:00p - 2:30p Session 2 - Cloud Software -An Introduction to Open Stack -Ian Foster, ANL and Univ. of Chicago, Globus Online -Best CCA11 Poster Talk 3:00p - 4:30p Session 3 - Cloud Applications -Matt Arrott, UCSD, Ocean Observatories Initiative -Alex Szalay, John Hopkins University, Simulation and Large Data -Robert Grossman, University of Chicago, Open Science Data Cloud 4:30p - 5:30p Panel and Discussion EXTENDED ABSTRACT TOPICS --------------------------------------------------------------------------------- * compute and storage cloud architectures and implementations * map-reduce and its generalizations * programming models and tools * novel data-intensive computing applications * data intensive scalable computing * distributed data intensive computing * content distribution systems for large data * data management within and across data centers * models, frameworks and systems for cloud security IMPORTANT DATES --------------------------------------------------------------------------------- Extended Abstract submission: March 15th, 2011 Acceptance notification: March 22nd, 2011 Final extended abstracts due: April 1st, 2011 Workshop date: April 12th-13th, 2011 EXTENDED ABSTRACT SUBMISSION --------------------------------------------------------------------------------- Authors are invited to submit extended abstracts of not more than 2 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages (including all text, figures, and references); please use the ACM 8.5 x 11 manuscript templates from http://www.acm.org/sigs/publications/proceedings-templates. A 150 word abstract and the final 2 page extended abstract (PDF format) must be submitted online at https://cmt.research.microsoft.com/CCA2011/ before the deadline of March 15th, 2011 at 11:59PM PST. The extended abstracts will be reviewed, and accepted abstracts will be published online at the CCA11 website (http://www.cca11.org/). Notifications of the paper decisions will be sent out by March 22nd, 2011. Submission implies the willingness of at least one of the authors to register and present the abstract in the poster session on April 12, 2011. One extended abstract will be chosen for a 30-minute presentation slot in the final program, which consists of invited leading researchers in Cloud Computing. For more information about the poster session, please visithttp://www.cca11.org/, or send questions to Ioan Raicu (iraicu at cs.iit.edu ). WORKSHOP CHAIRS --------------------------------------------------------------------------------- * Ian Foster, University of Chicago& Argonne National Laboratory * Bob Grossman, University of Chicago -- ================================================================= Ioan Raicu, Ph.D. Assistant Professor, Illinois Institute of Technology (IIT) Guest Research Faculty, Argonne National Laboratory (ANL) ================================================================= Data-Intensive Distributed Systems Laboratory, CS/IIT Distributed Systems Laboratory, MCS/ANL ================================================================= Cel: 1-847-722-0876 Office: 1-312-567-5704 Email: iraicu at cs.iit.edu Web: http://www.cs.iit.edu/~iraicu/ Web: http://datasys.cs.iit.edu/ ================================================================= ================================================================= -------------- next part -------------- An HTML attachment was scrubbed... URL: From fedorov at bwh.harvard.edu Sat Mar 5 17:23:12 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 5 Mar 2011 18:23:12 -0500 Subject: [Swift-user] Errors compiling swift-0.92 Message-ID: Hi, I have the errors below trying to compile latest release branch on Ubuntu 10.04 x86_64 following the instructions on this page: http://www.ci.uchicago.edu/swift/downloads/index.php Any suggestions? delete.dependency.log.1: [delete] Deleting: /home/andrey/local/cog/dependency.log.dist [echo] [swift]: DIST [echo] [swift]: JARCOPY [copy] Copying 9 files to /home/andrey/local/cog/modules/swift/dist/swift-svn/lib [copy] Copying 10 files to /home/andrey/local/cog/modules/swift/dist/swift-svn/etc delete.jar: [echo] [swift]: DELETE.JAR (cog-swift-svn.jar) compile: [echo] [swift]: COMPILE [mkdir] Created dir: /home/andrey/local/cog/modules/swift/build [javac] Compiling 374 source files to /home/andrey/local/cog/modules/swift/build [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:69: generics are not supported in -source 1.4 [javac] (use -source 5 or higher to enable generics) [javac] private List profiles; [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:163: for-each loops are not supported in -source 1.4 [javac] (use -source 5 or higher to enable for-each loops) [javac] for (Profile p : profiles) [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:394: annotations are not supported in -source 1.4 [javac] (use -source 5 or higher to enable annotations) [javac] @SuppressWarnings("hiding") [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalog.java:53: generics are not supported in -source 1.4 [javac] (use -source 5 or higher to enable generics) [javac] List getTCEntries( String namespace, String name, String version, [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:71: generics are not supported in -source 1.4 [javac] (use -source 5 or higher to enable generics) [javac] private List profiles; [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:165: for-each loops are not supported in -source 1.4 [javac] (use -source 5 or higher to enable for-each loops) [javac] for (Profile p : profiles) [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:80: generics are not supported in -source 1.4 [javac] (use -source 5 or higher to enable generics) [javac] private Map>> mTreeMap; [javac] ^ [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:465: for-each loops are not supported in -source 1.4 [javac] (use -source 5 or higher to enable for-each loops) [javac] for (Map> lMap : lfnList) { [... many more ...] -- Andriy Fedorov, Ph.D. Research Fellow Brigham and Women's Hospital Harvard Medical School 75 Francis Street Boston, MA 02115 USA fedorov at bwh.harvard.edu (617) 525-6258 (office) From hategan at mcs.anl.gov Sat Mar 5 19:06:47 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 05 Mar 2011 17:06:47 -0800 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: References: Message-ID: <1299373607.30442.1.camel@blabla2.none> Odd. In cog/mbuild.xml, towards the top, there should be a line reading: Do you have that? On Sat, 2011-03-05 at 18:23 -0500, Andriy Fedorov wrote: > Hi, > > I have the errors below trying to compile latest release branch on > Ubuntu 10.04 x86_64 following the instructions on this page: > http://www.ci.uchicago.edu/swift/downloads/index.php > > Any suggestions? > > delete.dependency.log.1: > [delete] Deleting: /home/andrey/local/cog/dependency.log.dist > [echo] [swift]: DIST > [echo] [swift]: JARCOPY > [copy] Copying 9 files to > /home/andrey/local/cog/modules/swift/dist/swift-svn/lib > [copy] Copying 10 files to > /home/andrey/local/cog/modules/swift/dist/swift-svn/etc > > delete.jar: > [echo] [swift]: DELETE.JAR (cog-swift-svn.jar) > > compile: > [echo] [swift]: COMPILE > [mkdir] Created dir: /home/andrey/local/cog/modules/swift/build > [javac] Compiling 374 source files to > /home/andrey/local/cog/modules/swift/build > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:69: > generics are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable generics) > [javac] private List profiles; > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:163: > for-each loops are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable for-each loops) > [javac] for (Profile p : profiles) > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:394: > annotations are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable annotations) > [javac] @SuppressWarnings("hiding") > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalog.java:53: > generics are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable generics) > [javac] List getTCEntries( String namespace, String > name, String version, > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:71: > generics are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable generics) > [javac] private List profiles; > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:165: > for-each loops are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable for-each loops) > [javac] for (Profile p : profiles) > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:80: > generics are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable generics) > [javac] private Map>> mTreeMap; > [javac] ^ > [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:465: > for-each loops are not supported in -source 1.4 > [javac] (use -source 5 or higher to enable for-each loops) > [javac] for (Map> lMap : lfnList) { > > [... many more ...] > > -- > Andriy Fedorov, Ph.D. > > Research Fellow > Brigham and Women's Hospital > Harvard Medical School > 75 Francis Street > Boston, MA 02115 USA > fedorov at bwh.harvard.edu > (617) 525-6258 (office) > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From fedorov at bwh.harvard.edu Sat Mar 5 19:10:09 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 5 Mar 2011 20:10:09 -0500 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: <1299373607.30442.1.camel@blabla2.none> References: <1299373607.30442.1.camel@blabla2.none> Message-ID: On Sat, Mar 5, 2011 at 20:06, Mihael Hategan wrote: > Odd. In cog/mbuild.xml, towards the top, there should be a line reading: > > > > Do you have that? > I have If I change that to 1.5, I still have errors ... > > On Sat, 2011-03-05 at 18:23 -0500, Andriy Fedorov wrote: >> Hi, >> >> I have the errors below trying to compile latest release branch on >> Ubuntu 10.04 x86_64 following the instructions on this page: >> http://www.ci.uchicago.edu/swift/downloads/index.php >> >> Any suggestions? >> >> delete.dependency.log.1: >> ? ?[delete] Deleting: /home/andrey/local/cog/dependency.log.dist >> ? ? ?[echo] [swift]: DIST >> ? ? ?[echo] [swift]: JARCOPY >> ? ? ?[copy] Copying 9 files to >> /home/andrey/local/cog/modules/swift/dist/swift-svn/lib >> ? ? ?[copy] Copying 10 files to >> /home/andrey/local/cog/modules/swift/dist/swift-svn/etc >> >> delete.jar: >> ? ? ?[echo] [swift]: DELETE.JAR (cog-swift-svn.jar) >> >> compile: >> ? ? ?[echo] [swift]: COMPILE >> ? ? [mkdir] Created dir: /home/andrey/local/cog/modules/swift/build >> ? ? [javac] Compiling 374 source files to >> /home/andrey/local/cog/modules/swift/build >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:69: >> generics are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable generics) >> ? ? [javac] ? ? private List profiles; >> ? ? [javac] ? ? ? ? ? ? ? ? ^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:163: >> for-each loops are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable for-each loops) >> ? ? [javac] ? ? ? ? ? ? for (Profile p : profiles) >> ? ? [javac] ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TCEntry.java:394: >> annotations are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable annotations) >> ? ? [javac] ? ? @SuppressWarnings("hiding") >> ? ? [javac] ? ? ?^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalog.java:53: >> generics are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable generics) >> ? ? [javac] ? ? List getTCEntries( String namespace, String >> name, String version, >> ? ? [javac] ? ? ? ? ^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:71: >> generics are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable generics) >> ? ? [javac] ? ? private List profiles; >> ? ? [javac] ? ? ? ? ? ? ? ? ^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/TransformationCatalogEntry.java:165: >> for-each loops are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable for-each loops) >> ? ? [javac] ? ? ? ? ? ? for (Profile p : profiles) >> ? ? [javac] ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:80: >> generics are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable generics) >> ? ? [javac] ? ? private Map>> mTreeMap; >> ? ? [javac] ? ? ? ? ? ? ? ?^ >> ? ? [javac] /home/andrey/local/cog/modules/swift/src/org/globus/swift/catalog/transformation/File.java:465: >> for-each loops are not supported in -source 1.4 >> ? ? [javac] (use -source 5 or higher to enable for-each loops) >> ? ? [javac] ? ? ? ? for (Map> lMap : lfnList) { >> >> [... many more ...] >> >> -- >> Andriy Fedorov, Ph.D. >> >> Research Fellow >> Brigham and Women's Hospital >> Harvard Medical School >> 75 Francis Street >> Boston, MA 02115 USA >> fedorov at bwh.harvard.edu >> (617) 525-6258 (office) >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > From hategan at mcs.anl.gov Sat Mar 5 19:18:50 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 05 Mar 2011 17:18:50 -0800 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: References: <1299373607.30442.1.camel@blabla2.none> Message-ID: <1299374330.31100.0.camel@blabla2.none> On Sat, 2011-03-05 at 20:10 -0500, Andriy Fedorov wrote: > On Sat, Mar 5, 2011 at 20:06, Mihael Hategan wrote: > > Odd. In cog/mbuild.xml, towards the top, there should be a line reading: > > > > > > > > Do you have that? > > > > I have > > If I change that to 1.5, I still have errors ... Even odder. Can you try a clean checkout? Mihael From fedorov at bwh.harvard.edu Sat Mar 5 19:25:50 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 5 Mar 2011 20:25:50 -0500 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: <1299374330.31100.0.camel@blabla2.none> References: <1299373607.30442.1.camel@blabla2.none> <1299374330.31100.0.camel@blabla2.none> Message-ID: On Sat, Mar 5, 2011 at 20:18, Mihael Hategan wrote: > On Sat, 2011-03-05 at 20:10 -0500, Andriy Fedorov wrote: >> On Sat, Mar 5, 2011 at 20:06, Mihael Hategan wrote: >> > Odd. In cog/mbuild.xml, towards the top, there should be a line reading: >> > >> > >> > >> > Do you have that? >> > >> >> I have >> >> If I change that to 1.5, I still have errors ... > > Even odder. Can you try a clean checkout? > Yes, but I just checked it out couple of hours ago fresh! Are there any peculiarities with java on linux? [andrey at beat swift] javac -version javac 1.6.0_20 [andrey at beat swift] java -version java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9.7) (6b20-1.9.7-0ubuntu1~10.04.1) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) [andrey at beat swift] uname -a Linux beat 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC 2011 x86_64 GNU/Linux > Mihael > > From hategan at mcs.anl.gov Sat Mar 5 19:54:25 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 05 Mar 2011 17:54:25 -0800 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: References: <1299373607.30442.1.camel@blabla2.none> <1299374330.31100.0.camel@blabla2.none> Message-ID: <1299376465.31100.5.camel@blabla2.none> On Sat, 2011-03-05 at 20:25 -0500, Andriy Fedorov wrote: > On Sat, Mar 5, 2011 at 20:18, Mihael Hategan wrote: > > On Sat, 2011-03-05 at 20:10 -0500, Andriy Fedorov wrote: > >> On Sat, Mar 5, 2011 at 20:06, Mihael Hategan wrote: > >> > Odd. In cog/mbuild.xml, towards the top, there should be a line reading: > >> > > >> > > >> > > >> > Do you have that? > >> > > >> > >> I have > >> > >> If I change that to 1.5, I still have errors ... > > > > Even odder. Can you try a clean checkout? > > > > Yes, but I just checked it out couple of hours ago fresh! Are there > any peculiarities with java on linux? Not that I know of. Linux is the platform on which swift is most used. But now that I look at the instructions, I think they are wrong. The correct cog branch is 4.1.8, not 4.1.7. Mihael From fedorov at bwh.harvard.edu Sun Mar 6 08:01:31 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sun, 6 Mar 2011 09:01:31 -0500 Subject: [Swift-user] Errors compiling swift-0.92 In-Reply-To: <1299376465.31100.5.camel@blabla2.none> References: <1299373607.30442.1.camel@blabla2.none> <1299374330.31100.0.camel@blabla2.none> <1299376465.31100.5.camel@blabla2.none> Message-ID: On Sat, Mar 5, 2011 at 20:54, Mihael Hategan wrote: > But now that I look at the instructions, I think they are wrong. The > correct cog branch is 4.1.8, not 4.1.7. > Yes, this indeed was the problem. The instructions on the Swift web page need an update. Thank you for resolving this! Andrey From frank.wang04 at gmail.com Tue Mar 8 07:57:24 2011 From: frank.wang04 at gmail.com (Weiyang Wang) Date: Tue, 8 Mar 2011 21:57:24 +0800 Subject: [Swift-user] [New Swift User] Simple .sh file mapping and running Message-ID: Hello, I'm trying to map .sh files and directly execute them by sh myfile.sh but encounter some errors here. Could anyone help? ####Contents of test.swift#### type SHL {}; SHL shellarray[] ; /*app ( SHLout out )shellexecute ( SHL s ){ sh @filename(s) stdout=@filename(out); // Could anyone add comments for these lines ? I'm not sure whether an out file is necessary }*/ app simpleshellexec(SHL s){ sh @filename(s); // Is this app written correctly? } foreach s in shellarray { simpleshellexec(s); } #####Contents of mapper.sh##### #!/bin/bash echo "[0] 0.sh" echo "[1] 1.sh" ############################# I'm expecting 0.sh and 1.sh can be run under current directories. Any corrections/ suggestions from anyone? Thank you. Weiyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From ketancmaheshwari at gmail.com Thu Mar 10 09:03:56 2011 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Thu, 10 Mar 2011 09:03:56 -0600 Subject: [Swift-user] Swift Coasters on OSG Message-ID: Hello, I have been trying the Swift Coasters on OSG as an exercise from Allan's scripts on CI endpoints (communicado). The coasters services seems to start well. However, I get error when I submit the Swift workflow. Following are the details of what I am doing: 0. source /opt/osg-1.2.16/setup.sh ok. 1. ./mk_catalog.rb whitelist extenci contents of whitelist: SPRACE_osg-ce.sprace.org.br EOF Generates empty {worker, slave}.swift, tc.data, {gt2_osg, condor_osg, coaster_osg}.xml 2. ./start_services.sh 2 Seems to start service normally as per service-*.log files 3. swift -config swift.properties -sites.file coaster_osg.xml slave.swift To configure services to run in passive mode, no errors 4. swift -config swift.properties -sites.file condor_osg.xml worker.swift Request coaster jobs: no errors 5. swift -config swift.properties -sites.file coaster_osg.xml sleep.swift Submit workflow sleep.swift. I get error message: No service contacts available. Any clues on this error? Regards, Ketan -------------- next part -------------- An HTML attachment was scrubbed... URL: From aespinosa at cs.uchicago.edu Thu Mar 10 10:17:26 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 10 Mar 2011 10:17:26 -0600 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: References: Message-ID: Hello Ketan, Can you give the output of 'condor_status -const 'GlueSiteUniqueID == "SPRACE"'? also can you try it with "SPRACE__osg-ce.sprace.org.br" (two underscores). I made minor changes to the script so that the output is more parsable and can be passed to other utilities like Globus Online. 2011/3/10 Ketan Maheshwari : > Hello, > I have been trying the Swift Coasters on OSG as an exercise from Allan's > scripts on CI endpoints (communicado). > The coasters services seems to start well. However, I get error when I > submit the Swift workflow. Following are the details of what I am doing: > 0. source /opt/osg-1.2.16/setup.sh > ?? ? ok. > 1. ./mk_catalog.rb whitelist extenci > ?? ?contents of whitelist: > ?? ?SPRACE_osg-ce.sprace.org.br > ?? ?EOF > ?? ?Generates empty {worker, slave}.swift, tc.data, {gt2_osg, condor_osg, > coaster_osg}.xml > > 2. ./start_services.sh 2 > ?? ? Seems to start service normally as per service-*.log files > 3.?swift -config swift.properties -sites.file coaster_osg.xml slave.swift > ?? ? To configure services to run in passive mode, no errors > 4.?swift -config swift.properties -sites.file condor_osg.xml worker.swift > ?? ? Request coaster jobs: no errors > 5.?swift -config swift.properties -sites.file coaster_osg.xml sleep.swift > ?? ?Submit workflow sleep.swift. > I get error message: No service contacts available. > Any clues on this error? > Regards, > Ketan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From wilde at mcs.anl.gov Thu Mar 10 11:36:52 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 10 Mar 2011 11:36:52 -0600 (CST) Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: Message-ID: <1844425469.174165.1299778612215.JavaMail.root@zimbra.anl.gov> re: I get error message: No service contacts available. Ive seen this error but I cant recall at the moment what it means. Im in a workshop but will try to dig through my emails (and memory... ; ) Does anyone else on this list know this one? Searching the list archives, trying small test cases, and sending more of the log info around the error message all may help. - Mike ----- Original Message ----- Hello, I have been trying the Swift Coasters on OSG as an exercise from Allan's scripts on CI endpoints (communicado). The coasters services seems to start well. However, I get error when I submit the Swift workflow. Following are the details of what I am doing: 0. source /opt/osg-1.2.16/setup.sh ok. 1. ./mk_catalog.rb whitelist extenci contents of whitelist: SPRACE_osg-ce.sprace.org.br EOF Generates empty {worker, slave}.swift, tc.data, {gt2_osg, condor_osg, coaster_osg}.xml 2. ./start_services.sh 2 Seems to start service normally as per service-*.log files 3. swift -config swift.properties -sites.file coaster_osg.xml slave.swift To configure services to run in passive mode, no errors 4. swift -config swift.properties -sites.file condor_osg.xml worker.swift Request coaster jobs: no errors 5. swift -config swift.properties -sites.file coaster_osg.xml sleep.swift Submit workflow sleep.swift. I get error message: No service contacts available. Any clues on this error? Regards, Ketan _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From hategan at mcs.anl.gov Thu Mar 10 12:01:18 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 10 Mar 2011 10:01:18 -0800 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: References: Message-ID: <1299780078.28297.1.camel@blabla2.none> It probably means your sites.xml is empty. Can you post it? Mihael On Thu, 2011-03-10 at 09:03 -0600, Ketan Maheshwari wrote: > > Submit workflow sleep.swift. > > > I get error message: No service contacts available. > From ketancmaheshwari at gmail.com Thu Mar 10 14:14:59 2011 From: ketancmaheshwari at gmail.com (Ketan Maheshwari) Date: Thu, 10 Mar 2011 14:14:59 -0600 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: <1299780078.28297.1.camel@blabla2.none> References: <1299780078.28297.1.camel@blabla2.none> Message-ID: Mihael, Yes, in fact sites.xml was actually missing. May be that is the reason for the error. However, Now, when I try to restart the coaster services using the attached start_services.sh script, I get the following error stack: Error starting coaster service: Address already in use Error starting coaster service java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365) at java.net.ServerSocket.bind(ServerSocket.java:319) at java.net.ServerSocket.(ServerSocket.java:185) at org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:114) at org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:68) at org.globus.net.BaseServer.(BaseServer.java:84) at org.globus.cog.karajan.workflow.service.GSSService.(GSSService.java:63) at org.globus.cog.abstraction.coaster.service.CoasterService.(CoasterService.java:74) at org.globus.cog.abstraction.coaster.service.CoasterPersistentService.(CoasterPersistentService.java:34) at org.globus.cog.abstraction.coaster.service.CoasterPersistentService.main(CoasterPersistentService.java:98) Perhaps, I need to kill the previously started service but couldn't figure how. Thanks, Ketan On Thu, Mar 10, 2011 at 12:01 PM, Mihael Hategan wrote: > It probably means your sites.xml is empty. Can you post it? > > Mihael > > On Thu, 2011-03-10 at 09:03 -0600, Ketan Maheshwari wrote: > > > > > Submit workflow sleep.swift. > > > > > > I get error message: No service contacts available. > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: start_services.sh Type: application/x-sh Size: 357 bytes Desc: not available URL: From hategan at mcs.anl.gov Thu Mar 10 14:19:29 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 10 Mar 2011 12:19:29 -0800 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: References: <1299780078.28297.1.camel@blabla2.none> Message-ID: <1299788369.349.1.camel@blabla2.none> On Thu, 2011-03-10 at 14:14 -0600, Ketan Maheshwari wrote: > Mihael, [...] > Error starting coaster service: Address already in use [...] > Perhaps, I need to kill the previously started service but couldn't > figure how. That or use it. kill ? Mihael From aespinosa at cs.uchicago.edu Thu Mar 10 14:28:47 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 10 Mar 2011 14:28:47 -0600 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: References: <1299780078.28297.1.camel@blabla2.none> Message-ID: Errr. I think that was my fault. I hardwired port numbers for coaster services and i'm running a coaster service at the moment. Clearly it needs to be fixed. -Allan 2011/3/10 Ketan Maheshwari : > Mihael, > Yes, in fact sites.xml was actually missing. May be that is the reason for > the error. > However, Now, when I try to restart the coaster services using the attached > start_services.sh script, I get the following error stack: > Error starting coaster service: Address already in use > Error starting coaster service > java.net.BindException: Address already in use > at java.net.PlainSocketImpl.socketBind(Native Method) > at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365) > at java.net.ServerSocket.bind(ServerSocket.java:319) > at java.net.ServerSocket.(ServerSocket.java:185) > at > org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:114) > at > org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:68) > at org.globus.net.BaseServer.(BaseServer.java:84) > at > org.globus.cog.karajan.workflow.service.GSSService.(GSSService.java:63) > at > org.globus.cog.abstraction.coaster.service.CoasterService.(CoasterService.java:74) > at > org.globus.cog.abstraction.coaster.service.CoasterPersistentService.(CoasterPersistentService.java:34) > at > org.globus.cog.abstraction.coaster.service.CoasterPersistentService.main(CoasterPersistentService.java:98) > Perhaps, I need to kill the previously started service but couldn't figure > how. > Thanks, > Ketan > > On Thu, Mar 10, 2011 at 12:01 PM, Mihael Hategan > wrote: >> >> It probably means your sites.xml is empty. Can you post it? >> >> Mihael >> >> On Thu, 2011-03-10 at 09:03 -0600, Ketan Maheshwari wrote: >> >> > >> > ? ? Submit workflow sleep.swift. >> > >> > >> > I get error message: No service contacts available. >> > >> >> From aespinosa at cs.uchicago.edu Thu Mar 10 14:30:07 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 10 Mar 2011 14:30:07 -0600 Subject: [Swift-user] Swift Coasters on OSG In-Reply-To: References: <1299780078.28297.1.camel@blabla2.none> Message-ID: Oh ignore that. (just saw your start_services.sh script) 2011/3/10 Allan Espinosa : > Errr. ?I think that was my fault. ?I hardwired port numbers for > coaster services and i'm running a coaster service at the moment. > > Clearly it needs to be fixed. > > -Allan > > 2011/3/10 Ketan Maheshwari : >> Mihael, >> Yes, in fact sites.xml was actually missing. May be that is the reason for >> the error. >> However, Now, when I try to restart the coaster services using the attached >> start_services.sh script, I get the following error stack: >> Error starting coaster service: Address already in use >> Error starting coaster service >> java.net.BindException: Address already in use >> at java.net.PlainSocketImpl.socketBind(Native Method) >> at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365) >> at java.net.ServerSocket.bind(ServerSocket.java:319) >> at java.net.ServerSocket.(ServerSocket.java:185) >> at >> org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:114) >> at >> org.globus.net.ServerSocketFactory.createServerSocket(ServerSocketFactory.java:68) >> at org.globus.net.BaseServer.(BaseServer.java:84) >> at >> org.globus.cog.karajan.workflow.service.GSSService.(GSSService.java:63) >> at >> org.globus.cog.abstraction.coaster.service.CoasterService.(CoasterService.java:74) >> at >> org.globus.cog.abstraction.coaster.service.CoasterPersistentService.(CoasterPersistentService.java:34) >> at >> org.globus.cog.abstraction.coaster.service.CoasterPersistentService.main(CoasterPersistentService.java:98) >> Perhaps, I need to kill the previously started service but couldn't figure >> how. >> Thanks, >> Ketan >> >> On Thu, Mar 10, 2011 at 12:01 PM, Mihael Hategan >> wrote: >>> >>> It probably means your sites.xml is empty. Can you post it? >>> >>> Mihael >>> >>> On Thu, 2011-03-10 at 09:03 -0600, Ketan Maheshwari wrote: >>> >>> > >>> > ? ? Submit workflow sleep.swift. >>> > >>> > >>> > I get error message: No service contacts available. From aespinosa at cs.uchicago.edu Fri Mar 11 14:06:41 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 11 Mar 2011 14:06:41 -0600 Subject: [Swift-user] 3rd party transfer optimization Message-ID: Hi, I know you can pass 3rd party optimizations like tcp buffer size to low-level Karajan provider definitions. Does anyone have examples on applying these to Swift's sites.xml definitions? Thanks, -Allan From mickelso at mcs.anl.gov Sat Mar 12 07:00:42 2011 From: mickelso at mcs.anl.gov (Sheri Mickelson) Date: Sat, 12 Mar 2011 07:00:42 -0600 Subject: [Swift-user] Adding :bigmem to node count Message-ID: <4D7B6E7A.5090502@mcs.anl.gov> Hi, To run a job without swift I need to specify "-l nodes=2:bigmem" in either my qsub command or add it to my run script. How do I tell swift to add ":bigmem" to the node total? Thanks, Sheri From wilde at mcs.anl.gov Sat Mar 12 10:52:19 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sat, 12 Mar 2011 10:52:19 -0600 (CST) Subject: [Swift-user] Adding :bigmem to node count In-Reply-To: <4D7B6E7A.5090502@mcs.anl.gov> Message-ID: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> Sheri, I am not yet sure, but I think we'll need to add a feature in Swift to enable this specification to be passed down to qsub. That should be doable fairly soon; I had to do something similar to run on Cray XE/XT systems, which I did based on Swift 0.92. What machine is this for, and which Swift release or build are you running? In the meantime, a solution you could use more immediately would be to run with manual/persistent coasters, where you could specify your own qsub. This works right now but requires some wrapper scripting that is not yet packaged or sufficiently documented, so thats a route that also requires some work from you and/or us. I know Justin is working with you, so we should all coordinate and hopefully we can resolve this soon for you. - Mike ----- Original Message ----- > Hi, > > To run a job without swift I need to specify "-l nodes=2:bigmem" in > either my qsub command or add it > to my run script. How do I tell swift to add ":bigmem" to the node > total? > > Thanks, Sheri > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From mickelso at mcs.anl.gov Sat Mar 12 11:01:30 2011 From: mickelso at mcs.anl.gov (Sheri Mickelson) Date: Sat, 12 Mar 2011 11:01:30 -0600 Subject: [Swift-user] Adding :bigmem to node count In-Reply-To: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> References: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> Message-ID: <4D7BA6EA.4000700@mcs.anl.gov> Hi Mike, It's for fusion and I'm running with swift-r3967. Thanks, Sheri Michael Wilde wrote: > Sheri, I am not yet sure, but I think we'll need to add a feature in Swift to enable this specification to be passed down to qsub. That should be doable fairly soon; I had to do something similar to run on Cray XE/XT systems, which I did based on Swift 0.92. > > What machine is this for, and which Swift release or build are you running? > > In the meantime, a solution you could use more immediately would be to run with manual/persistent coasters, where you could specify your own qsub. This works right now but requires some wrapper scripting that is not yet packaged or sufficiently documented, so thats a route that also requires some work from you and/or us. > > I know Justin is working with you, so we should all coordinate and hopefully we can resolve this soon for you. > > - Mike > > ----- Original Message ----- >> Hi, >> >> To run a job without swift I need to specify "-l nodes=2:bigmem" in >> either my qsub command or add it >> to my run script. How do I tell swift to add ":bigmem" to the node >> total? >> >> Thanks, Sheri >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > From wilde at mcs.anl.gov Sun Mar 13 11:22:13 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Sun, 13 Mar 2011 11:22:13 -0500 (CDT) Subject: [Swift-user] 3rd party transfer optimization In-Reply-To: Message-ID: <1905494453.183707.1300033333505.JavaMail.root@zimbra.anl.gov> Allan, this ability does not exist to my knowledge, but sounds useful. (Its similar in nature to Sheri's request for adding qsub attributes). Have you verified (ie by hardcoding the desired value) that it will improve the data transfer rates? If so, and if you can add it, that would be great. Are you running off of trunk or a branch? Lets move discussion to swift-devel to discuss how this would work and where to make the change. - Mike ----- Original Message ----- > Hi, > > I know you can pass 3rd party optimizations like tcp buffer size to > low-level Karajan provider definitions. Does anyone have examples on > applying these to Swift's sites.xml definitions? > > Thanks, > -Allan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From hategan at mcs.anl.gov Sun Mar 13 13:24:09 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sun, 13 Mar 2011 11:24:09 -0700 Subject: [Swift-user] 3rd party transfer optimization In-Reply-To: <1905494453.183707.1300033333505.JavaMail.root@zimbra.anl.gov> References: <1905494453.183707.1300033333505.JavaMail.root@zimbra.anl.gov> Message-ID: <1300040649.17758.6.camel@blabla2.none> A long time ago I added into Karajan the ability to set gridftp buffer options using a file that was used at the time by TGCP. That feature should still be available and can probably be integrated transparently. I can probably take a shot at it today. Mihael On Sun, 2011-03-13 at 11:22 -0500, Michael Wilde wrote: > Allan, this ability does not exist to my knowledge, but sounds useful. > > (Its similar in nature to Sheri's request for adding qsub attributes). > > Have you verified (ie by hardcoding the desired value) that it will improve the data transfer rates? > > If so, and if you can add it, that would be great. > Are you running off of trunk or a branch? > > Lets move discussion to swift-devel to discuss how this would work and where to make the change. > > - Mike > > > ----- Original Message ----- > > Hi, > > > > I know you can pass 3rd party optimizations like tcp buffer size to > > low-level Karajan provider definitions. Does anyone have examples on > > applying these to Swift's sites.xml definitions? > > > > Thanks, > > -Allan > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > From wozniak at mcs.anl.gov Sun Mar 13 17:17:55 2011 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Sun, 13 Mar 2011 17:17:55 -0500 (Central Daylight Time) Subject: [Swift-user] Adding :bigmem to node count In-Reply-To: <4D7BA6EA.4000700@mcs.anl.gov> References: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> <4D7BA6EA.4000700@mcs.anl.gov> Message-ID: I'll take a look at this- it will probably be a small CoG feature addition. On Sat, 12 Mar 2011, Sheri Mickelson wrote: > Hi Mike, > > It's for fusion and I'm running with swift-r3967. > > Thanks, Sheri > > Michael Wilde wrote: >> Sheri, I am not yet sure, but I think we'll need to add a feature in Swift >> to enable this specification to be passed down to qsub. That should be >> doable fairly soon; I had to do something similar to run on Cray XE/XT >> systems, which I did based on Swift 0.92. >> >> What machine is this for, and which Swift release or build are you running? >> >> In the meantime, a solution you could use more immediately would be to run >> with manual/persistent coasters, where you could specify your own qsub. >> This works right now but requires some wrapper scripting that is not yet >> packaged or sufficiently documented, so thats a route that also requires >> some work from you and/or us. >> >> I know Justin is working with you, so we should all coordinate and >> hopefully we can resolve this soon for you. >> >> - Mike >> >> ----- Original Message ----- >>> Hi, >>> >>> To run a job without swift I need to specify "-l nodes=2:bigmem" in >>> either my qsub command or add it >>> to my run script. How do I tell swift to add ":bigmem" to the node >>> total? >>> >>> Thanks, Sheri >>> _______________________________________________ >>> Swift-user mailing list >>> Swift-user at ci.uchicago.edu >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > -- Justin M Wozniak From fedorov at bwh.harvard.edu Mon Mar 14 10:06:11 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 14 Mar 2011 11:06:11 -0400 Subject: [Swift-user] Swift is stuck with 5K jobs Message-ID: Hi, I am using swift with coasters on NCSA Abe. I use binary build of swift 0.92. My script should generate about 5K individual jobs. When I try to run it, I have Swift svn swift-r4157 cog-r3056 RunID: 20110314-0951-f3c45zja Progress: Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "SIGINT handler" Exception in thread "SIGINT handler" Exception in thread "SIGTERM handler" After this error, I am not able to terminate the script, and no jobs get scheduled to pbs apparently. Am I hitting some limit? Is 5K jobs too much? How do I terminate swift now not to waste cycles of the head node? Thanks -- Andriy Fedorov, Ph.D. Research Fellow Brigham and Women's Hospital Harvard Medical School 75 Francis Street Boston, MA 02115 USA fedorov at bwh.harvard.edu (617) 525-6258 (office) From aespinosa at cs.uchicago.edu Mon Mar 14 10:15:57 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Mon, 14 Mar 2011 10:15:57 -0500 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: References: Message-ID: Hello Andriy, The default package may have a small max heap limit. Usually, I apply this patch whenever I get a new version of Swift: --- old/bin/swift 2010-10-12 12:18:47.000000000 -0500 +++ new/bin/swift 2010-10-12 12:18:37.000000000 -0500 @@ -9,7 +9,7 @@ CYGWIN= CPDELIM=":" -HEAPMAX=256M +HEAPMAX=4096M if echo `uname` | grep -i "cygwin"; then CYGWIN="yes" Works well with 800K jobs. -Allan 2011/3/14 Andriy Fedorov : > Hi, > > I am using swift with coasters on NCSA Abe. I use binary build of > swift 0.92. My script should generate about 5K individual jobs. When I > try to run it, I have > > Swift svn swift-r4157 cog-r3056 > > RunID: 20110314-0951-f3c45zja > Progress: > Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space > > Exception in thread "SIGINT handler" > Exception in thread "SIGINT handler" Exception in thread "SIGTERM handler" > > After this error, I am not able to terminate the script, and no jobs > get scheduled to pbs apparently. > > Am I hitting some limit? Is 5K jobs too much? > > How do I terminate swift now not to waste cycles of the head node? > > Thanks > -- > Andriy Fedorov, Ph.D. > > Research Fellow > Brigham and Women's Hospital > Harvard Medical School > 75 Francis Street > Boston, MA 02115 USA > fedorov at bwh.harvard.edu > (617) 525-6258 (office) > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > -- Allan M. Espinosa PhD student, Computer Science University of Chicago From fedorov at bwh.harvard.edu Mon Mar 14 10:28:59 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 14 Mar 2011 11:28:59 -0400 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: References: Message-ID: Thanks, Allan. Now I have a different exception: class org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer throws exception in doStuff. Fix it! java.lang.NullPointerException at org.globus.cog.abstraction.impl.file.coaster.commands.PutFileCommand.error(PutFileCommand.java:95) at org.globus.cog.abstraction.impl.file.coaster.buffers.ReadBuffer.error(ReadBuffer.java:79) at org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer.doStuff(NIOChannelReadBuffer.java:42) at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:133) On Mon, Mar 14, 2011 at 11:15, Allan Espinosa wrote: > Hello Andriy, > > The default package may have a small max heap limit. ?Usually, I apply > this patch whenever I get a new version of Swift: > > --- old/bin/swift ? ? ? 2010-10-12 12:18:47.000000000 -0500 > +++ new/bin/swift ? ? ? 2010-10-12 12:18:37.000000000 -0500 > @@ -9,7 +9,7 @@ > > ?CYGWIN= > ?CPDELIM=":" > -HEAPMAX=256M > +HEAPMAX=4096M > > ?if echo `uname` | grep -i "cygwin"; then > ? CYGWIN="yes" > > > Works well with 800K jobs. > > -Allan > > 2011/3/14 Andriy Fedorov : >> Hi, >> >> I am using swift with coasters on NCSA Abe. I use binary build of >> swift 0.92. My script should generate about 5K individual jobs. When I >> try to run it, I have >> >> Swift svn swift-r4157 cog-r3056 >> >> RunID: 20110314-0951-f3c45zja >> Progress: >> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space >> >> Exception in thread "SIGINT handler" >> Exception in thread "SIGINT handler" Exception in thread "SIGTERM handler" >> >> After this error, I am not able to terminate the script, and no jobs >> get scheduled to pbs apparently. >> >> Am I hitting some limit? Is 5K jobs too much? >> >> How do I terminate swift now not to waste cycles of the head node? >> >> Thanks >> -- >> Andriy Fedorov, Ph.D. >> >> Research Fellow >> Brigham and Women's Hospital >> Harvard Medical School >> 75 Francis Street >> Boston, MA 02115 USA >> fedorov at bwh.harvard.edu >> (617) 525-6258 (office) >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> >> > > > > -- > Allan M. Espinosa > PhD student, Computer Science > University of Chicago > From wilde at mcs.anl.gov Mon Mar 14 11:19:00 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 14 Mar 2011 11:19:00 -0500 (CDT) Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: Message-ID: <302148457.186130.1300119540971.JavaMail.root@zimbra.anl.gov> Hi Andriy, Can you post your sites.xml, tc, properties (if you are changing any) and swift command line? It looks to me like perhaps your script and configuration is trying to use provider staging with coasters. Or perhaps the coaster data provider. Was that intended? (I say this because of the "PutFileCommand" method listed in the traceback. Mihael, is that for provider staging or coaster data provider?) If so, can you try this with a local file provider, is using something like this in sites.xml: /home/yourhomedir/swiftwork If thats already what you have, then Im not sure whats happening. If you were trying one of the coaster-based data transfer methods, we need to dig deeper into whats failing but hopefully the local data provider will get you further for now. - Mike ----- Original Message ----- > Thanks, Allan. Now I have a different exception: > > class > org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer > throws exception in doStuff. Fix it! > java.lang.NullPointerException > at > org.globus.cog.abstraction.impl.file.coaster.commands.PutFileCommand.error(PutFileCommand.java:95) > at > org.globus.cog.abstraction.impl.file.coaster.buffers.ReadBuffer.error(ReadBuffer.java:79) > at > org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer.doStuff(NIOChannelReadBuffer.java:42) > at > org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:133) > > > > On Mon, Mar 14, 2011 at 11:15, Allan Espinosa > wrote: > > Hello Andriy, > > > > The default package may have a small max heap limit. Usually, I > > apply > > this patch whenever I get a new version of Swift: > > > > --- old/bin/swift 2010-10-12 12:18:47.000000000 -0500 > > +++ new/bin/swift 2010-10-12 12:18:37.000000000 -0500 > > @@ -9,7 +9,7 @@ > > > > ?CYGWIN= > > ?CPDELIM=":" > > -HEAPMAX=256M > > +HEAPMAX=4096M > > > > ?if echo `uname` | grep -i "cygwin"; then > > ? CYGWIN="yes" > > > > > > Works well with 800K jobs. > > > > -Allan > > > > 2011/3/14 Andriy Fedorov : > >> Hi, > >> > >> I am using swift with coasters on NCSA Abe. I use binary build of > >> swift 0.92. My script should generate about 5K individual jobs. > >> When I > >> try to run it, I have > >> > >> Swift svn swift-r4157 cog-r3056 > >> > >> RunID: 20110314-0951-f3c45zja > >> Progress: > >> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap > >> space > >> > >> Exception in thread "SIGINT handler" > >> Exception in thread "SIGINT handler" Exception in thread "SIGTERM > >> handler" > >> > >> After this error, I am not able to terminate the script, and no > >> jobs > >> get scheduled to pbs apparently. > >> > >> Am I hitting some limit? Is 5K jobs too much? > >> > >> How do I terminate swift now not to waste cycles of the head node? > >> > >> Thanks > >> -- > >> Andriy Fedorov, Ph.D. > >> > >> Research Fellow > >> Brigham and Women's Hospital > >> Harvard Medical School > >> 75 Francis Street > >> Boston, MA 02115 USA > >> fedorov at bwh.harvard.edu > >> (617) 525-6258 (office) > >> _______________________________________________ > >> Swift-user mailing list > >> Swift-user at ci.uchicago.edu > >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > >> > >> > > > > > > > > -- > > Allan M. Espinosa > > PhD student, Computer Science > > University of Chicago > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From hategan at mcs.anl.gov Mon Mar 14 12:26:53 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 14 Mar 2011 10:26:53 -0700 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: References: Message-ID: <1300123613.15580.0.camel@blabla2.none> The whole log would probably help here. Mihael On Mon, 2011-03-14 at 11:28 -0400, Andriy Fedorov wrote: > Thanks, Allan. Now I have a different exception: > > class org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer > throws exception in doStuff. Fix it! > java.lang.NullPointerException > at org.globus.cog.abstraction.impl.file.coaster.commands.PutFileCommand.error(PutFileCommand.java:95) > at org.globus.cog.abstraction.impl.file.coaster.buffers.ReadBuffer.error(ReadBuffer.java:79) > at org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer.doStuff(NIOChannelReadBuffer.java:42) > at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:133) > > > > On Mon, Mar 14, 2011 at 11:15, Allan Espinosa wrote: > > Hello Andriy, > > > > The default package may have a small max heap limit. Usually, I apply > > this patch whenever I get a new version of Swift: > > > > --- old/bin/swift 2010-10-12 12:18:47.000000000 -0500 > > +++ new/bin/swift 2010-10-12 12:18:37.000000000 -0500 > > @@ -9,7 +9,7 @@ > > > > CYGWIN= > > CPDELIM=":" > > -HEAPMAX=256M > > +HEAPMAX=4096M > > > > if echo `uname` | grep -i "cygwin"; then > > CYGWIN="yes" > > > > > > Works well with 800K jobs. > > > > -Allan > > > > 2011/3/14 Andriy Fedorov : > >> Hi, > >> > >> I am using swift with coasters on NCSA Abe. I use binary build of > >> swift 0.92. My script should generate about 5K individual jobs. When I > >> try to run it, I have > >> > >> Swift svn swift-r4157 cog-r3056 > >> > >> RunID: 20110314-0951-f3c45zja > >> Progress: > >> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space > >> > >> Exception in thread "SIGINT handler" > >> Exception in thread "SIGINT handler" Exception in thread "SIGTERM handler" > >> > >> After this error, I am not able to terminate the script, and no jobs > >> get scheduled to pbs apparently. > >> > >> Am I hitting some limit? Is 5K jobs too much? > >> > >> How do I terminate swift now not to waste cycles of the head node? > >> > >> Thanks > >> -- > >> Andriy Fedorov, Ph.D. > >> > >> Research Fellow > >> Brigham and Women's Hospital > >> Harvard Medical School > >> 75 Francis Street > >> Boston, MA 02115 USA > >> fedorov at bwh.harvard.edu > >> (617) 525-6258 (office) > >> _______________________________________________ > >> Swift-user mailing list > >> Swift-user at ci.uchicago.edu > >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > >> > >> > > > > > > > > -- > > Allan M. Espinosa > > PhD student, Computer Science > > University of Chicago > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From hategan at mcs.anl.gov Mon Mar 14 12:29:34 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 14 Mar 2011 10:29:34 -0700 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: References: Message-ID: <1300123774.15580.1.camel@blabla2.none> On Mon, 2011-03-14 at 11:06 -0400, Andriy Fedorov wrote: > Am I hitting some limit? Is 5K jobs too much? Shouldn't be, but if you have the coaster service running in local mode, that might do the trick. > > How do I terminate swift now not to waste cycles of the head node? kill -9 From fedorov at bwh.harvard.edu Mon Mar 14 12:43:08 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 14 Mar 2011 13:43:08 -0400 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: <302148457.186130.1300119540971.JavaMail.root@zimbra.anl.gov> References: <302148457.186130.1300119540971.JavaMail.root@zimbra.anl.gov> Message-ID: Hi all, Thank you for your help! I indeed had coaster data provider. After reporting the original problem, I switched back to an earlier version of swift, and I observed some very strange errors -- the input files arrived corrupted. Once I changed the data provider to local, the data corruption problem seems to have disappeared. I have not tried yet to use 0.92 again, will let you know when I do if I run into problems. Sorry I cannot set aside time to debug this, trying to get the actual work done ... AF On Mon, Mar 14, 2011 at 12:19, Michael Wilde wrote: > Hi Andriy, > > Can you post your sites.xml, tc, properties (if you are changing any) and swift command line? > > It looks to me like perhaps your script and configuration is trying to use provider staging with coasters. Or perhaps the coaster data provider. Was that intended? (I say this because of the "PutFileCommand" method listed in the traceback. Mihael, is that for provider staging or coaster data provider?) > > If so, can you try this with a local file provider, is using something like this in sites.xml: > > ? ? > ? ?/home/yourhomedir/swiftwork > > If thats already what you have, then Im not sure whats happening. > > If you were trying one of the coaster-based data transfer methods, we need to dig deeper into whats failing but hopefully the local data provider will get you further for now. > > - Mike > > > ----- Original Message ----- >> Thanks, Allan. Now I have a different exception: >> >> class >> org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer >> throws exception in doStuff. Fix it! >> java.lang.NullPointerException >> at >> org.globus.cog.abstraction.impl.file.coaster.commands.PutFileCommand.error(PutFileCommand.java:95) >> at >> org.globus.cog.abstraction.impl.file.coaster.buffers.ReadBuffer.error(ReadBuffer.java:79) >> at >> org.globus.cog.abstraction.impl.file.coaster.buffers.NIOChannelReadBuffer.doStuff(NIOChannelReadBuffer.java:42) >> at >> org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:133) >> >> >> >> On Mon, Mar 14, 2011 at 11:15, Allan Espinosa >> wrote: >> > Hello Andriy, >> > >> > The default package may have a small max heap limit. Usually, I >> > apply >> > this patch whenever I get a new version of Swift: >> > >> > --- old/bin/swift 2010-10-12 12:18:47.000000000 -0500 >> > +++ new/bin/swift 2010-10-12 12:18:37.000000000 -0500 >> > @@ -9,7 +9,7 @@ >> > >> > ?CYGWIN= >> > ?CPDELIM=":" >> > -HEAPMAX=256M >> > +HEAPMAX=4096M >> > >> > ?if echo `uname` | grep -i "cygwin"; then >> > ? CYGWIN="yes" >> > >> > >> > Works well with 800K jobs. >> > >> > -Allan >> > >> > 2011/3/14 Andriy Fedorov : >> >> Hi, >> >> >> >> I am using swift with coasters on NCSA Abe. I use binary build of >> >> swift 0.92. My script should generate about 5K individual jobs. >> >> When I >> >> try to run it, I have >> >> >> >> Swift svn swift-r4157 cog-r3056 >> >> >> >> RunID: 20110314-0951-f3c45zja >> >> Progress: >> >> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap >> >> space >> >> >> >> Exception in thread "SIGINT handler" >> >> Exception in thread "SIGINT handler" Exception in thread "SIGTERM >> >> handler" >> >> >> >> After this error, I am not able to terminate the script, and no >> >> jobs >> >> get scheduled to pbs apparently. >> >> >> >> Am I hitting some limit? Is 5K jobs too much? >> >> >> >> How do I terminate swift now not to waste cycles of the head node? >> >> >> >> Thanks >> >> -- >> >> Andriy Fedorov, Ph.D. >> >> >> >> Research Fellow >> >> Brigham and Women's Hospital >> >> Harvard Medical School >> >> 75 Francis Street >> >> Boston, MA 02115 USA >> >> fedorov at bwh.harvard.edu >> >> (617) 525-6258 (office) >> >> _______________________________________________ >> >> Swift-user mailing list >> >> Swift-user at ci.uchicago.edu >> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> >> >> >> >> > >> > >> > >> > -- >> > Allan M. Espinosa >> > PhD student, Computer Science >> > University of Chicago >> > >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > From wilde at mcs.anl.gov Mon Mar 14 12:45:09 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 14 Mar 2011 12:45:09 -0500 (CDT) Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: <1300123774.15580.1.camel@blabla2.none> Message-ID: <666136219.186760.1300124709728.JavaMail.root@zimbra.anl.gov> Andriy, All, On systems like TeraGrid hosts where the login hosts are frequently heavily loaded, we should verify that you can obtain a single interactive compute node via qsub -I on which to run the swift command (ideally under screen to make re-attachment easy) and that from there Swift can run jobs using the Coaster-over-PBS provider configuration. I suspect (and hope) that any cluster node on say abe, queenbee, and ranger can also run qsub and qstat. We should test and document that, but in the meantime, Andriy, can you try that approach? I *think* that it should be identical to running from a login host. What I want to avoid is causing too heavy a load on any login host and in the process getting Swift banned or having it associated with causing system problems. Thanks and regards, - Mike ----- Original Message ----- > On Mon, 2011-03-14 at 11:06 -0400, Andriy Fedorov wrote: > > Am I hitting some limit? Is 5K jobs too much? > > Shouldn't be, but if you have the coaster service running in local > mode, > that might do the trick. > > > > > How do I terminate swift now not to waste cycles of the head node? > > kill -9 > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From fedorov at bwh.harvard.edu Mon Mar 14 13:30:47 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 14 Mar 2011 14:30:47 -0400 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: <666136219.186760.1300124709728.JavaMail.root@zimbra.anl.gov> References: <1300123774.15580.1.camel@blabla2.none> <666136219.186760.1300124709728.JavaMail.root@zimbra.anl.gov> Message-ID: Michael, This is a very good observation. The problem is one has to know approximately how long the total run of the swift script will take, which includes the time to wait in the queue for the computing resources. I do not know how such estimations can be reliably obtained. IMHO, submission from the head node is ok, since it occupies only one CPU. However, I believe processes that are running on the head node for more than 30 minutes are terminated automatically, so submission from the head node may not work for all cases. Any other ideas? -- Andriy Fedorov, Ph.D. Research Fellow Brigham and Women's Hospital Harvard Medical School 75 Francis Street Boston, MA 02115 USA fedorov at bwh.harvard.edu (617) 525-6258 (office) On Mon, Mar 14, 2011 at 13:45, Michael Wilde wrote: > Andriy, All, > > On systems like TeraGrid hosts where the login hosts are frequently heavily loaded, we should verify that you can obtain a single interactive compute node via qsub -I on which to run the swift command (ideally under screen to make re-attachment easy) and that from there Swift can run jobs using the Coaster-over-PBS provider configuration. > > I suspect (and hope) that any cluster node on say abe, queenbee, and ranger can also run qsub and qstat. ?We should test and document that, but in the meantime, Andriy, can you try that approach? I *think* that it should be identical to running from a login host. > > What I want to avoid is causing too heavy a load on any login host and in the process getting Swift banned or having it associated with causing system problems. > > Thanks and regards, > > - Mike > > > ----- Original Message ----- >> On Mon, 2011-03-14 at 11:06 -0400, Andriy Fedorov wrote: >> > Am I hitting some limit? Is 5K jobs too much? >> >> Shouldn't be, but if you have the coaster service running in local >> mode, >> that might do the trick. >> >> > >> > How do I terminate swift now not to waste cycles of the head node? >> >> kill -9 >> >> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > From hategan at mcs.anl.gov Mon Mar 14 14:04:17 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 14 Mar 2011 12:04:17 -0700 Subject: [Swift-user] 3rd party transfer optimization In-Reply-To: <1300040649.17758.6.camel@blabla2.none> References: <1905494453.183707.1300033333505.JavaMail.root@zimbra.anl.gov> <1300040649.17758.6.camel@blabla2.none> Message-ID: <1300129457.21562.2.camel@blabla2.none> This should be in in trunk swift r4176 cog r3062. What you need to mess with is a file in etc called bpd.conf. It's lines are in the form: source, destination -> bufferSize This file was (and may still be) in the standard format used by TGCP, so presumably you can fetch it from an existing TGCP installation. However, it is already provided in swift, but may not be up to date. Please test and let me know if it works. Mihael On Sun, 2011-03-13 at 11:24 -0700, Mihael Hategan wrote: > A long time ago I added into Karajan the ability to set gridftp buffer > options using a file that was used at the time by TGCP. That feature > should still be available and can probably be integrated transparently. > I can probably take a shot at it today. > > Mihael > > On Sun, 2011-03-13 at 11:22 -0500, Michael Wilde wrote: > > Allan, this ability does not exist to my knowledge, but sounds useful. > > > > (Its similar in nature to Sheri's request for adding qsub attributes). > > > > Have you verified (ie by hardcoding the desired value) that it will improve the data transfer rates? > > > > If so, and if you can add it, that would be great. > > Are you running off of trunk or a branch? > > > > Lets move discussion to swift-devel to discuss how this would work and where to make the change. > > > > - Mike > > > > > > ----- Original Message ----- > > > Hi, > > > > > > I know you can pass 3rd party optimizations like tcp buffer size to > > > low-level Karajan provider definitions. Does anyone have examples on > > > applying these to Swift's sites.xml definitions? > > > > > > Thanks, > > > -Allan > > > _______________________________________________ > > > Swift-user mailing list > > > Swift-user at ci.uchicago.edu > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From wilde at mcs.anl.gov Mon Mar 14 15:49:24 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Mon, 14 Mar 2011 15:49:24 -0500 (CDT) Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: Message-ID: <1052687223.187990.1300135764368.JavaMail.root@zimbra.anl.gov> Andriy, Another alternative is to run Swift outside of the cluster - eg on a script execution host at your home institution - and manually start the coaster workers in a PBS job. These workers would connect back to the swift command (or to an external coaster server process) to pick up jobs to run. This takes some scripting and documentation that is not provided in the Swift release yet, but its a strategy that we could help you with if and when needed. I think another feature of Swift, Collective Data Management (CDM), can be used in such cases to specify that your data files exist on the cluster side rather than on the host that's running the swift command. - Mike ----- Original Message ----- > Michael, > > This is a very good observation. > > The problem is one has to know approximately how long the total run of > the swift script will take, which includes the time to wait in the > queue for the computing resources. I do not know how such estimations > can be reliably obtained. > > IMHO, submission from the head node is ok, since it occupies only one > CPU. However, I believe processes that are running on the head node > for more than 30 minutes are terminated automatically, so submission > from the head node may not work for all cases. > > Any other ideas? > > -- > Andriy Fedorov, Ph.D. > > Research Fellow > Brigham and Women's Hospital > Harvard Medical School > 75 Francis Street > Boston, MA 02115 USA > fedorov at bwh.harvard.edu > (617) 525-6258 (office) > > > > On Mon, Mar 14, 2011 at 13:45, Michael Wilde > wrote: > > Andriy, All, > > > > On systems like TeraGrid hosts where the login hosts are frequently > > heavily loaded, we should verify that you can obtain a single > > interactive compute node via qsub -I on which to run the swift > > command (ideally under screen to make re-attachment easy) and that > > from there Swift can run jobs using the Coaster-over-PBS provider > > configuration. > > > > I suspect (and hope) that any cluster node on say abe, queenbee, and > > ranger can also run qsub and qstat. We should test and document > > that, but in the meantime, Andriy, can you try that approach? I > > *think* that it should be identical to running from a login host. > > > > What I want to avoid is causing too heavy a load on any login host > > and in the process getting Swift banned or having it associated with > > causing system problems. > > > > Thanks and regards, > > > > - Mike > > > > > > ----- Original Message ----- > >> On Mon, 2011-03-14 at 11:06 -0400, Andriy Fedorov wrote: > >> > Am I hitting some limit? Is 5K jobs too much? > >> > >> Shouldn't be, but if you have the coaster service running in local > >> mode, > >> that might do the trick. > >> > >> > > >> > How do I terminate swift now not to waste cycles of the head > >> > node? > >> > >> kill -9 > >> > >> > >> _______________________________________________ > >> Swift-user mailing list > >> Swift-user at ci.uchicago.edu > >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > > -- > > Michael Wilde > > Computation Institute, University of Chicago > > Mathematics and Computer Science Division > > Argonne National Laboratory > > > > -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From fedorov at bwh.harvard.edu Mon Mar 14 16:20:52 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Mon, 14 Mar 2011 17:20:52 -0400 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: <1052687223.187990.1300135764368.JavaMail.root@zimbra.anl.gov> References: <1052687223.187990.1300135764368.JavaMail.root@zimbra.anl.gov> Message-ID: On Mon, Mar 14, 2011 at 16:49, Michael Wilde wrote: > but its a strategy that we could help you with if and when needed. > Good to know there is a strategy! >From the practical point of view, my approach will be to run from head node until someone slaps my hands, and if the job is terminated prematurely because it exceeds the wallclock limit on the head node, use Swift restart capability to continue. If someone puts together a documentation how to run it from a remote location, I might try it. So I understand this was not a problem so far? Does this mean no-one ever ran a swift script that was taking more than 30 minutes to complete, including queue time? Just curious if there is another workaround I am not aware of. > I think another feature of Swift, Collective Data Management (CDM), can be used in such cases to specify that your data files exist on the cluster side rather than > on the host that's running the swift command. > > - Mike > > ----- Original Message ----- >> Michael, >> >> This is a very good observation. >> >> The problem is one has to know approximately how long the total run of >> the swift script will take, which includes the time to wait in the >> queue for the computing resources. I do not know how such estimations >> can be reliably obtained. >> >> IMHO, submission from the head node is ok, since it occupies only one >> CPU. However, I believe processes that are running on the head node >> for more than 30 minutes are terminated automatically, so submission >> from the head node may not work for all cases. >> >> Any other ideas? >> >> -- >> Andriy Fedorov, Ph.D. >> >> Research Fellow >> Brigham and Women's Hospital >> Harvard Medical School >> 75 Francis Street >> Boston, MA 02115 USA >> fedorov at bwh.harvard.edu >> (617) 525-6258 (office) >> >> >> >> On Mon, Mar 14, 2011 at 13:45, Michael Wilde >> wrote: >> > Andriy, All, >> > >> > On systems like TeraGrid hosts where the login hosts are frequently >> > heavily loaded, we should verify that you can obtain a single >> > interactive compute node via qsub -I on which to run the swift >> > command (ideally under screen to make re-attachment easy) and that >> > from there Swift can run jobs using the Coaster-over-PBS provider >> > configuration. >> > >> > I suspect (and hope) that any cluster node on say abe, queenbee, and >> > ranger can also run qsub and qstat. We should test and document >> > that, but in the meantime, Andriy, can you try that approach? I >> > *think* that it should be identical to running from a login host. >> > >> > What I want to avoid is causing too heavy a load on any login host >> > and in the process getting Swift banned or having it associated with >> > causing system problems. >> > >> > Thanks and regards, >> > >> > - Mike >> > >> > >> > ----- Original Message ----- >> >> On Mon, 2011-03-14 at 11:06 -0400, Andriy Fedorov wrote: >> >> > Am I hitting some limit? Is 5K jobs too much? >> >> >> >> Shouldn't be, but if you have the coaster service running in local >> >> mode, >> >> that might do the trick. >> >> >> >> > >> >> > How do I terminate swift now not to waste cycles of the head >> >> > node? >> >> >> >> kill -9 >> >> >> >> >> >> _______________________________________________ >> >> Swift-user mailing list >> >> Swift-user at ci.uchicago.edu >> >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > >> > -- >> > Michael Wilde >> > Computation Institute, University of Chicago >> > Mathematics and Computer Science Division >> > Argonne National Laboratory >> > >> > > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > From hategan at mcs.anl.gov Mon Mar 14 19:32:26 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Mon, 14 Mar 2011 17:32:26 -0700 Subject: [Swift-user] Swift is stuck with 5K jobs In-Reply-To: <1052687223.187990.1300135764368.JavaMail.root@zimbra.anl.gov> References: <1052687223.187990.1300135764368.JavaMail.root@zimbra.anl.gov> Message-ID: <1300149146.22737.1.camel@blabla2.none> On Mon, 2011-03-14 at 15:49 -0500, Michael Wilde wrote: > Andriy, > > Another alternative is to run Swift outside of the cluster - eg on a > script execution host at your home institution - and manually start > the coaster workers in a PBS job. These workers would connect back to > the swift command (or to an external coaster server process) to pick > up jobs to run. Given that he's using coasters in automated mode, it may be easier to keep using it in automated mode but submit through GRAM and keep only the coaster service on the head node while swift would be running on some local machine which would not have as many restrictions. Mihael From wozniak at mcs.anl.gov Fri Mar 18 14:16:37 2011 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Fri, 18 Mar 2011 14:16:37 -0500 (CDT) Subject: [Swift-user] Adding :bigmem to node count In-Reply-To: References: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> <4D7BA6EA.4000700@mcs.anl.gov> Message-ID: Hello Actually, it looks like there may be a queue for bigmem jobs, is that what you want to use? If so, you would just modify the queue profile entry in sites.xml . Justin On Sun, 13 Mar 2011, Justin M Wozniak wrote: > > I'll take a look at this- it will probably be a small CoG feature addition. > > On Sat, 12 Mar 2011, Sheri Mickelson wrote: > >> Hi Mike, >> >> It's for fusion and I'm running with swift-r3967. >> >> Thanks, Sheri >> >> Michael Wilde wrote: >>> Sheri, I am not yet sure, but I think we'll need to add a feature in Swift >>> to enable this specification to be passed down to qsub. That should be >>> doable fairly soon; I had to do something similar to run on Cray XE/XT >>> systems, which I did based on Swift 0.92. >>> >>> What machine is this for, and which Swift release or build are you >>> running? >>> >>> In the meantime, a solution you could use more immediately would be to run >>> with manual/persistent coasters, where you could specify your own qsub. >>> This works right now but requires some wrapper scripting that is not yet >>> packaged or sufficiently documented, so thats a route that also requires >>> some work from you and/or us. >>> >>> I know Justin is working with you, so we should all coordinate and >>> hopefully we can resolve this soon for you. >>> >>> - Mike >>> >>> ----- Original Message ----- >>>> Hi, >>>> >>>> To run a job without swift I need to specify "-l nodes=2:bigmem" in >>>> either my qsub command or add it >>>> to my run script. How do I tell swift to add ":bigmem" to the node >>>> total? >>>> >>>> Thanks, Sheri >>>> _______________________________________________ >>>> Swift-user mailing list >>>> Swift-user at ci.uchicago.edu >>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >>> >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >> > > -- Justin M Wozniak From fedorov at bwh.harvard.edu Sat Mar 19 21:36:30 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sat, 19 Mar 2011 22:36:30 -0400 Subject: [Swift-user] Continue after failure Message-ID: Hi, In my execution scenario, I would like to allow the swift script to continue even if some of the individual jobs failed. Is it possible to tell swift to try to execute the rest of the jobs, if a failure of some jobs is detected? Andrey From hategan at mcs.anl.gov Sat Mar 19 22:16:54 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Sat, 19 Mar 2011 20:16:54 -0700 Subject: [Swift-user] Continue after failure In-Reply-To: References: Message-ID: <1300591014.10403.1.camel@blabla2.none> It's called lazy errors and you enable it by saying swift -lazy.errors true ... or by setting the respective property in the swift configuration file. Mihael On Sat, 2011-03-19 at 22:36 -0400, Andriy Fedorov wrote: > Hi, > > In my execution scenario, I would like to allow the swift script to > continue even if some of the individual jobs failed. > > Is it possible to tell swift to try to execute the rest of the jobs, > if a failure of some jobs is detected? > > Andrey > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user From fedorov at bwh.harvard.edu Sun Mar 20 08:58:07 2011 From: fedorov at bwh.harvard.edu (Andriy Fedorov) Date: Sun, 20 Mar 2011 09:58:07 -0400 Subject: [Swift-user] Continue after failure In-Reply-To: <1300591014.10403.1.camel@blabla2.none> References: <1300591014.10403.1.camel@blabla2.none> Message-ID: Thanks, Mihael -- this is exactly what I needed! On Sat, Mar 19, 2011 at 23:16, Mihael Hategan wrote: > It's called lazy errors and you enable it by saying > > swift -lazy.errors true ... > > or by setting the respective property in the swift configuration file. > > Mihael > > On Sat, 2011-03-19 at 22:36 -0400, Andriy Fedorov wrote: >> Hi, >> >> In my execution scenario, I would like to allow the swift script to >> continue even if some of the individual jobs failed. >> >> Is it possible to tell swift to try to execute the rest of the jobs, >> if a failure of some jobs is detected? >> >> Andrey >> _______________________________________________ >> Swift-user mailing list >> Swift-user at ci.uchicago.edu >> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > From wozniak at mcs.anl.gov Mon Mar 21 11:29:43 2011 From: wozniak at mcs.anl.gov (Justin M Wozniak) Date: Mon, 21 Mar 2011 11:29:43 -0500 (CDT) Subject: [Swift-user] Adding :bigmem to node count In-Reply-To: References: <652847923.182956.1299948739399.JavaMail.root@zimbra.anl.gov> <4D7BA6EA.4000700@mcs.anl.gov> Message-ID: Hello Ok, I do not think that the queue is actually there (I found that in an old web page.) There is a new feature in trunk to support this. I tested the sites.xml file pasted below and it did give me a fat node on Fusion (as reported by /usr/bin/free). Justin 5 bigmem /home/wozniak/work On Fri, 18 Mar 2011, Justin M Wozniak wrote: > Hello > Actually, it looks like there may be a queue for bigmem jobs, is that > what you want to use? If so, you would just modify the queue profile entry > in sites.xml . > Justin > > On Sun, 13 Mar 2011, Justin M Wozniak wrote: > >> >> I'll take a look at this- it will probably be a small CoG feature addition. >> >> On Sat, 12 Mar 2011, Sheri Mickelson wrote: >> >>> Hi Mike, >>> >>> It's for fusion and I'm running with swift-r3967. >>> >>> Thanks, Sheri >>> >>> Michael Wilde wrote: >>>> Sheri, I am not yet sure, but I think we'll need to add a feature in >>>> Swift to enable this specification to be passed down to qsub. That should >>>> be doable fairly soon; I had to do something similar to run on Cray XE/XT >>>> systems, which I did based on Swift 0.92. >>>> >>>> What machine is this for, and which Swift release or build are you >>>> running? >>>> >>>> In the meantime, a solution you could use more immediately would be to >>>> run with manual/persistent coasters, where you could specify your own >>>> qsub. This works right now but requires some wrapper scripting that is >>>> not yet packaged or sufficiently documented, so thats a route that also >>>> requires some work from you and/or us. >>>> >>>> I know Justin is working with you, so we should all coordinate and >>>> hopefully we can resolve this soon for you. >>>> >>>> - Mike >>>> >>>> ----- Original Message ----- >>>>> Hi, >>>>> >>>>> To run a job without swift I need to specify "-l nodes=2:bigmem" in >>>>> either my qsub command or add it >>>>> to my run script. How do I tell swift to add ":bigmem" to the node >>>>> total? >>>>> >>>>> Thanks, Sheri >>>>> _______________________________________________ >>>>> Swift-user mailing list >>>>> Swift-user at ci.uchicago.edu >>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >>>> >>> _______________________________________________ >>> Swift-user mailing list >>> Swift-user at ci.uchicago.edu >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user >>> >> >> > > -- Justin M Wozniak From jon.monette at gmail.com Wed Mar 23 12:24:54 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Wed, 23 Mar 2011 12:24:54 -0500 Subject: [Swift-user] print statement Message-ID: I am using the current trunk. What is the function that allows to print to the screen? It used to be tracef but now swift is telling me that this is an unknown function. -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.monette at gmail.com Wed Mar 23 12:55:33 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Wed, 23 Mar 2011 12:55:33 -0500 Subject: [Swift-user] Re: print statement In-Reply-To: References: Message-ID: Nevermind. I was doing @tracef. It was complaining about the '@' so I dropped it. On Wed, Mar 23, 2011 at 12:24 PM, Jonathan Monette wrote: > I am using the current trunk. What is the function that allows to print to > the screen? It used to be tracef but now swift is telling me that this is > an unknown function. > > -- > Any intelligent fool can make things bigger and more complex... It takes a > touch of genius - and a lot of courage to move in the opposite direction. > - Albert Einstein > > > -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.monette at gmail.com Wed Mar 23 18:18:44 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Wed, 23 Mar 2011 18:18:44 -0500 Subject: [Swift-user] sites file question Message-ID: I have this line in my sites.xml file. 10 I want to do something like this ${MONTAGE_WORKERS} Is this possible or another way for this? The reason is depending on the data set i want a different number of workers. Some sets are large and I need to request 100 workers and others are small that 5 will suffice. So I would like to be able to set an environment variable in my run script detailing how many workers to set here. I am using automatic coasters(not passive or persistent). I would like to avoid manual coasters as automatic coasters is working fine for my jobs. I would like to do this so I can only keep one sites file and not a sites file for small, medium, and large data sets. -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Wed Mar 23 21:18:52 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Wed, 23 Mar 2011 21:18:52 -0500 (CDT) Subject: [Swift-user] sites file question In-Reply-To: Message-ID: <686432341.21278.1300933132884.JavaMail.root@zimbra.anl.gov> Since you have a run script, you can have it generate the sites file from a shell "here document" in which you can place shell variables, which will get substituted when the here document is referenced. David's gew gensites command does this, so you can likely use gensites itself from your run script, with your own sites file templates. if you need to, you can pass the same variables into your Swift script via the -param=value parameters at the end of the command line. - Mike ----- Original Message ----- I have this line in my sites.xml file. 10 I want to do something like this ${MONTAGE_WORKERS} Is this possible or another way for this? The reason is depending on the data set i want a different number of workers. Some sets are large and I need to request 100 workers and others are small that 5 will suffice. So I would like to be able to set an environment variable in my run script detailing how many workers to set here. I am using automatic coasters(not passive or persistent). I would like to avoid manual coasters as automatic coasters is working fine for my jobs. I would like to do this so I can only keep one sites file and not a sites file for small, medium, and large data sets. -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein _______________________________________________ Swift-user mailing list Swift-user at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.monette at gmail.com Wed Mar 23 21:48:14 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Wed, 23 Mar 2011 21:48:14 -0500 Subject: [Swift-user] sites file question In-Reply-To: <686432341.21278.1300933132884.JavaMail.root@zimbra.anl.gov> References: <686432341.21278.1300933132884.JavaMail.root@zimbra.anl.gov> Message-ID: Ok. thanks. I'll take a look at that. On Wed, Mar 23, 2011 at 9:18 PM, Michael Wilde wrote: > Since you have a run script, you can have it generate the sites file from a > shell "here document" in which you can place shell variables, which will get > substituted when the here document is referenced. > > David's gew gensites command does this, so you can likely use gensites > itself from your run script, with your own sites file templates. > > if you need to, you can pass the same variables into your Swift script via > the -param=value parameters at the end of the command line. > > - Mike > > ------------------------------ > > I have this line in my sites.xml file. > 10 > > I want to do something like this > ${MONTAGE_WORKERS} > > Is this possible or another way for this? The reason is depending on the > data set i want a different number of workers. Some sets are large and I > need to request 100 workers and others are small that 5 will suffice. So I > would like to be able to set an environment variable in my run script > detailing how many workers to set here. I am using automatic coasters(not > passive or persistent). I would like to avoid manual coasters as automatic > coasters is working fine for my jobs. I would like to do this so I can only > keep one sites file and not a sites file for small, medium, and large data > sets. > -- > Any intelligent fool can make things bigger and more complex... It takes a > touch of genius - and a lot of courage to move in the opposite direction. > - Albert Einstein > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > > > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From frank.wang04 at gmail.com Thu Mar 24 12:59:31 2011 From: frank.wang04 at gmail.com (Weiyang Wang) Date: Thu, 24 Mar 2011 12:59:31 -0500 Subject: [Swift-user] SWIFT sites choosing Message-ID: Hello, I've tested my SWIFT codes on login nodes of PADS and want to utilize more than 2 cores. Where can I find sites.xml on PADS? Is there one existing? I wrote up my own tc.data too, specifying running programs on "localhost". If I want to utilize the resources of PADS what should I changed it to? Thank you. Weiyang -------------- next part -------------- An HTML attachment was scrubbed... URL: From tianyu491433909 at 163.com Sun Mar 20 04:05:42 2011 From: tianyu491433909 at 163.com (tianyu491433909) Date: Sun, 20 Mar 2011 17:05:42 +0800 (CST) Subject: [Swift-user] a swift question Message-ID: <675b30da.c910.12ed28349f0.Coremail.tianyu491433909@163.com> Dear Mr/Ms: I'm doingworkon theswift, pleasehelp mesolve aproblem.May I askhow swiftis runningthatscript has afew nodes,wherethe code thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From glen842 at uchicago.edu Thu Mar 24 13:12:37 2011 From: glen842 at uchicago.edu (Glen Hocky) Date: Thu, 24 Mar 2011 14:12:37 -0400 Subject: [Swift-user] SWIFT sites choosing In-Reply-To: References: Message-ID: Something like this: where internal host name is the ip of the node you are submitting from (the one below is login2.pads I think) node granularity is how many jobs you want batched together and ppn going from 1-8 specifies how many cores you want per job you are running just set maxnodes to how many jobs you want submitted to the queue at once and jobthrottle=(maxnodes-1)/100 fill in PROJECT, SOMETHING and as desired 12:00:00 1 1 172.5.86.6 1 250--> 2.49 10000 PROJECT /tmp SOMETHING/swiftwork On Thu, Mar 24, 2011 at 1:59 PM, Weiyang Wang wrote: > Hello, > > I've tested my SWIFT codes on login nodes of PADS and want to utilize more > than 2 cores. > > Where can I find sites.xml on PADS? Is there one existing? > > I wrote up my own tc.data too, specifying running programs on "localhost". > If I want to utilize the resources of PADS what should I changed it to? > > Thank you. > > Weiyang > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aespinosa at cs.uchicago.edu Thu Mar 24 16:28:51 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 24 Mar 2011 16:28:51 -0500 Subject: [Swift-user] determining unmapped paths Message-ID: I'm trying figure out where in my workflow is causing this problem: 2011-03-24 16:23:50,485-0500 WARN FlowNode Ex098 java.lang.IllegalStateException: mapper.existing() returned a path [3] that it cannot subsequently map at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:129) at org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:67) at org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:53) at org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:80) at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:583) at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:396) at org.griphyn.vdl.mapping.ArrayDataNode.closeDeep(ArrayDataNode.java:51) at org.griphyn.vdl.karajan.lib.PartialCloseDataset.function(PartialCloseDataset.java:79) at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) It doesn't specify which data object it crashes on so i'm quite clueless at this point. I'm using the latest trunk any particular log4j class i should be enabling to debug? Thanks, -Allan -- Allan M. Espinosa PhD student, Computer Science University of Chicago From hategan at mcs.anl.gov Thu Mar 24 20:32:16 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Thu, 24 Mar 2011 18:32:16 -0700 Subject: [Swift-user] determining unmapped paths In-Reply-To: References: Message-ID: <1301016736.12059.5.camel@blabla2.none> This looks like an internal problem. Mappers should be two way functions, so they should be able to map all things returned by existing(). If you call D the subset of the set of swift data for a mapper and F corresponding subset of filesystem objects, map is a function from D to F, while existing returns a collection of objects in D. So theoretically map on each element of that collection should work. In this case the object in D is [3], which indicates that we are probably talking about an array mapper. Could you tell me which one it is? And also what files you are trying to map with it. Mihael On Thu, 2011-03-24 at 16:28 -0500, Allan Espinosa wrote: > I'm trying figure out where in my workflow is causing this problem: > > 2011-03-24 16:23:50,485-0500 WARN FlowNode Ex098 > java.lang.IllegalStateException: mapper.existing() returned a path [3] > that it cannot subsequently map > at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:129) > at org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:67) > at org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:53) > at org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:80) > at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:583) > at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:396) > at org.griphyn.vdl.mapping.ArrayDataNode.closeDeep(ArrayDataNode.java:51) > at org.griphyn.vdl.karajan.lib.PartialCloseDataset.function(PartialCloseDataset.java:79) > at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) > at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) > at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > It doesn't specify which data object it crashes on so i'm quite > clueless at this point. > > I'm using the latest trunk > > any particular log4j class i should be enabling to debug? > > > Thanks, > -Allan > From wilde at mcs.anl.gov Thu Mar 24 22:51:01 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 24 Mar 2011 22:51:01 -0500 (CDT) Subject: [Swift-user] determining unmapped paths In-Reply-To: <1301016736.12059.5.camel@blabla2.none> Message-ID: <1838581217.25851.1301025061847.JavaMail.root@zimbra.anl.gov> Possible a mismatch between an external mapper's output and the object that its mapping? I.e., wrong dimensionality, or a structure field mismatch? Im guessing [3] in this case was the string returned by the mapper, but Im not sure. (I say that based on the many reports of similar errors in the pas on the list, some from you, Allan). - Mike ----- Original Message ----- > This looks like an internal problem. Mappers should be two way > functions, so they should be able to map all things returned by > existing(). If you call D the subset of the set of swift data for a > mapper and F corresponding subset of filesystem objects, map is a > function from D to F, while existing returns a collection of objects > in > D. So theoretically map on each element of that collection should > work. > > In this case the object in D is [3], which indicates that we are > probably talking about an array mapper. Could you tell me which one it > is? And also what files you are trying to map with it. > > Mihael > > On Thu, 2011-03-24 at 16:28 -0500, Allan Espinosa wrote: > > I'm trying figure out where in my workflow is causing this problem: > > > > 2011-03-24 16:23:50,485-0500 WARN FlowNode Ex098 > > java.lang.IllegalStateException: mapper.existing() returned a path > > [3] > > that it cannot subsequently map > > at > > org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:129) > > at > > org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:67) > > at > > org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:53) > > at > > org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:80) > > at > > org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:583) > > at > > org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:396) > > at > > org.griphyn.vdl.mapping.ArrayDataNode.closeDeep(ArrayDataNode.java:51) > > at > > org.griphyn.vdl.karajan.lib.PartialCloseDataset.function(PartialCloseDataset.java:79) > > at > > org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > > at > > org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > > at > > org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > > at > > org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > > at > > org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) > > at > > org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:619) > > > > It doesn't specify which data object it crashes on so i'm quite > > clueless at this point. > > > > I'm using the latest trunk > > > > any particular log4j class i should be enabling to debug? > > > > > > Thanks, > > -Allan > > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From aespinosa at cs.uchicago.edu Thu Mar 24 23:23:43 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 24 Mar 2011 23:23:43 -0500 Subject: [Swift-user] determining unmapped paths In-Reply-To: <1301016736.12059.5.camel@blabla2.none> References: <1301016736.12059.5.camel@blabla2.none> Message-ID: Well I have something like type file; string arrays[] = readData('afile'); file data[] ; 2011/3/24 Mihael Hategan : > This looks like an internal problem. Mappers should be two way > functions, so they should be able to map all things returned by > existing(). If you call D the subset of the set of swift data for a > mapper and F corresponding subset of filesystem objects, map is a > function from D to F, while existing returns a collection of objects in > D. So theoretically map on each element of that collection should work. > > In this case the object in D is [3], which indicates that we are > probably talking about an array mapper. Could you tell me which one it > is? And also what files you are trying to map with it. > > Mihael > > On Thu, 2011-03-24 at 16:28 -0500, Allan Espinosa wrote: >> I'm trying figure out where in my workflow is causing this problem: >> >> 2011-03-24 16:23:50,485-0500 WARN ?FlowNode Ex098 >> java.lang.IllegalStateException: mapper.existing() returned a path [3] >> that it cannot subsequently map >> ? ? ? ? at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:129) >> ? ? ? ? at org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:67) >> ? ? ? ? at org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:53) >> ? ? ? ? at org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:80) >> ? ? ? ? at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:583) >> ? ? ? ? at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:396) >> ? ? ? ? at org.griphyn.vdl.mapping.ArrayDataNode.closeDeep(ArrayDataNode.java:51) >> ? ? ? ? at org.griphyn.vdl.karajan.lib.PartialCloseDataset.function(PartialCloseDataset.java:79) >> ? ? ? ? at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) >> ? ? ? ? at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) >> ? ? ? ? at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) >> ? ? ? ? at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) >> ? ? ? ? at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) >> ? ? ? ? at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) >> ? ? ? ? at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) >> ? ? ? ? at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) >> ? ? ? ? at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> ? ? ? ? at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> ? ? ? ? at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> ? ? ? ? at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> ? ? ? ? at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> ? ? ? ? at java.lang.Thread.run(Thread.java:619) >> >> It doesn't specify which data object it crashes on so i'm quite >> clueless at this point. >> >> I'm using the latest trunk >> >> any particular log4j class i should be enabling to debug? From aespinosa at cs.uchicago.edu Thu Mar 24 23:30:40 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Thu, 24 Mar 2011 23:30:40 -0500 Subject: [Swift-user] Re: determining unmapped paths In-Reply-To: References: Message-ID: Oh btw, this only happens in trunk. My workflow gets over to stage-ins on the release-0.92 branch. 2011/3/24 Allan Espinosa : > I'm trying figure out where in my workflow is causing this problem: > > 2011-03-24 16:23:50,485-0500 WARN ?FlowNode Ex098 > java.lang.IllegalStateException: mapper.existing() returned a path [3] > that it cannot subsequently map > ? ? ? ?at org.griphyn.vdl.mapping.RootDataNode.checkInputs(RootDataNode.java:129) > ? ? ? ?at org.griphyn.vdl.mapping.RootArrayDataNode.checkInputs(RootArrayDataNode.java:67) > ? ? ? ?at org.griphyn.vdl.mapping.RootArrayDataNode.innerInit(RootArrayDataNode.java:53) > ? ? ? ?at org.griphyn.vdl.mapping.RootArrayDataNode.handleClosed(RootArrayDataNode.java:80) > ? ? ? ?at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:583) > ? ? ? ?at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:396) > ? ? ? ?at org.griphyn.vdl.mapping.ArrayDataNode.closeDeep(ArrayDataNode.java:51) > ? ? ? ?at org.griphyn.vdl.karajan.lib.PartialCloseDataset.function(PartialCloseDataset.java:79) > ? ? ? ?at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:68) > ? ? ? ?at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) > ? ? ? ?at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) > ? ? ? ?at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) > ? ? ? ?at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) > ? ? ? ?at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) > ? ? ? ?at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:104) > ? ? ? ?at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) > ? ? ? ?at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > ? ? ? ?at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > ? ? ? ?at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ? ? ? ?at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > ? ? ? ?at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > ? ? ? ?at java.lang.Thread.run(Thread.java:619) > > It doesn't specify which data object it crashes on so i'm quite > clueless at this point. > > I'm using the latest trunk > > any particular log4j class i should be enabling to debug? > > > Thanks, > -Allan From hategan at mcs.anl.gov Fri Mar 25 03:41:28 2011 From: hategan at mcs.anl.gov (Mihael Hategan) Date: Fri, 25 Mar 2011 01:41:28 -0700 Subject: [Swift-user] determining unmapped paths In-Reply-To: <1838581217.25851.1301025061847.JavaMail.root@zimbra.anl.gov> References: <1838581217.25851.1301025061847.JavaMail.root@zimbra.anl.gov> Message-ID: <1301042488.8445.2.camel@blabla2.none> On Thu, 2011-03-24 at 22:51 -0500, Michael Wilde wrote: > Possible a mismatch between an external mapper's output and the object > that its mapping? I.e., wrong dimensionality, or a structure field > mismatch? > > Im guessing [3] in this case was the string returned by the mapper, > but Im not sure. (I say that based on the many reports of similar > errors in the pas on the list, some from you, Allan). [3] means whatever array you had, the third element in it. It's not a string, it's a type "path" (though one can convert certain classes of strings to such type paths by virtue of the fact that one ultimately expresses such things in a textual language). From aespinosa at cs.uchicago.edu Fri Mar 25 13:49:20 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 25 Mar 2011 13:49:20 -0500 Subject: [Swift-user] array slicing? Message-ID: Are there plans to incorporate this into the language? I want to perform some aggregate operations in subelements of my array: type file; file somearray[] ; file reduction1; file reduction2; file reduction3; file commoninput <'somefile'>; reduction1 = some_reduction(somearray[0:60], commoninput); reduction2 = some_reduction(somearray[61:120], commoninput); reduction3 = some_reduction(somearray[120:n], commoninput); I am doing this so that I have more control over the number of times commoninput is being staged to multiple resources. -Allan From wilde at mcs.anl.gov Fri Mar 25 13:51:58 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Fri, 25 Mar 2011 13:51:58 -0500 (CDT) Subject: [Swift-user] array slicing? In-Reply-To: Message-ID: <1645571246.28045.1301079118289.JavaMail.root@zimbra.anl.gov> You could add this to the Swift GSoC page, Allan. - Mike ----- Original Message ----- > Are there plans to incorporate this into the language? > > I want to perform some aggregate operations in subelements of my > array: > > type file; > > file somearray[] ; > > > file reduction1; > file reduction2; > file reduction3; > file commoninput <'somefile'>; > > reduction1 = some_reduction(somearray[0:60], commoninput); > reduction2 = some_reduction(somearray[61:120], commoninput); > reduction3 = some_reduction(somearray[120:n], commoninput); > > > > I am doing this so that I have more control over the number of times > commoninput is being staged to multiple resources. > > > -Allan > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From aespinosa at cs.uchicago.edu Fri Mar 25 14:01:19 2011 From: aespinosa at cs.uchicago.edu (Allan Espinosa) Date: Fri, 25 Mar 2011 14:01:19 -0500 Subject: [Swift-user] array slicing? In-Reply-To: <1645571246.28045.1301079118289.JavaMail.root@zimbra.anl.gov> References: <1645571246.28045.1301079118289.JavaMail.root@zimbra.anl.gov> Message-ID: Hi Mike, I think this could be added as a line to 'enhancing the library' or Ben's 'functional iteration constructs' -Allan PS I don't have commit access to dev.globus as I'm not technically a Globus committer 2011/3/25 Michael Wilde : > You could add this to the Swift GSoC page, Allan. > > - Mike > > ----- Original Message ----- >> Are there plans to incorporate this into the language? >> >> I want to perform some aggregate operations in subelements of my >> array: >> >> type file; >> >> file somearray[] ; >> >> >> file reduction1; >> file reduction2; >> file reduction3; >> file commoninput <'somefile'>; >> >> reduction1 = some_reduction(somearray[0:60], commoninput); >> reduction2 = some_reduction(somearray[61:120], commoninput); >> reduction3 = some_reduction(somearray[120:n], commoninput); >> >> >> >> I am doing this so that I have more control over the number of times >> commoninput is being staged to multiple resources. >> >> >> -Allan From wilde at mcs.anl.gov Thu Mar 31 17:43:24 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 31 Mar 2011 17:43:24 -0500 (CDT) Subject: [Swift-user] Swift loops with no explanation when no pending jobs will fit into any possible coaster block In-Reply-To: Message-ID: <435908826.53072.1301611404604.JavaMail.root@zimbra.anl.gov> I want to point this out to users: if you run a script using coasters as your job execution provider, and you see Swift just saying something like this, even though you know your coasters are running: RunID: 20110331-1702-3kfa6xa3 Progress: Progress: Initializing site shared directory:1 Progress: Stage in:1 Progress: Submitted:1 Progress: Submitted:1 then the problem is that your app maxwalltime (likely from tc.data or a default) s larger than the maxtime (after adjustments) of your coaster blocks. - Mike ----- Forwarded Message ----- From: bugzilla-daemon at mcs.anl.gov To: swift-devel at ci.uchicago.edu Sent: Thursday, March 31, 2011 5:34:54 PM Subject: [Swift-devel] [Bug 287] New: Swift loops with no explanation when no pending jobs will fit into any possible coaster block https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=287 Summary: Swift loops with no explanation when no pending jobs will fit into any possible coaster block Product: Swift Version: 0.93 Platform: All OS/Version: All Status: NEW Severity: major Priority: P1 Component: SwiftScript language AssignedTo: hategan at mcs.anl.gov ReportedBy: wilde at mcs.anl.gov CC: hategan at mcs.anl.gov Example: tc entry is: localhost cat /bin/cat null null GLOBUS::maxwalltime="00:05:00" sites pool is: 1 1 1 1 120 100 100 0.00 10000 /home/wilde/swiftwork cat app declares need for 5 mins walltime only possible coaster slot is 2 mins walltime so Swift just loops with a job in the queue that never gets run: RunID: 20110331-1702-3kfa6xa3 Progress: Progress: Initializing site shared directory:1 Progress: Stage in:1 Progress: Submitted:1 Progress: Submitted:1 User never gets an error like "No coaster slots exist with sufficient time remaining to run your job. I think the coaster block times out for inactivity, another one starts, and nothing gets run, and the user is left in the dark as to why. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug. You are watching the reporter. _______________________________________________ Swift-devel mailing list Swift-devel at ci.uchicago.edu http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From jon.monette at gmail.com Thu Mar 31 20:49:19 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Thu, 31 Mar 2011 20:49:19 -0500 Subject: [Swift-user] gensites documentation Message-ID: Where is the gensites scripts documented? I want to use it to generate a pads-coaster implementation that I have written by hand to add into my run script but not sure how to use this. -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Thu Mar 31 21:01:34 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 31 Mar 2011 21:01:34 -0500 (CDT) Subject: [Swift-user] gensites documentation In-Reply-To: Message-ID: <154345204.53538.1301623294720.JavaMail.root@zimbra.anl.gov> ----- Original Message ----- > Where is the gensites scripts documented? https://sites.google.com/site/swiftguide/home/managingsites >I want to use it to generate > a pads-coaster implementation that I have written by hand to add into > my run script but not sure how to use this. See if you can figure out how to proceed from the doc above and the source code, and suggest what you think is a reasonable approach on swift-devel. We're discussing a few changes to gensites but want to move it forward to production and supported use. I noted somewhere that the -l option mentioned doesnt seem to be in the current code - not sure what the issue is. You can probably add new templates without much discussion. Ive suggested some kind of help test mechanism for each template and its parameters. - Mike > > -- > Any intelligent fool can make things bigger and more complex... It > takes a touch of genius - and a lot of courage to move in the opposite > direction. > - Albert Einstein > > > > _______________________________________________ > Swift-user mailing list > Swift-user at ci.uchicago.edu > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From jon.monette at gmail.com Thu Mar 31 21:04:12 2011 From: jon.monette at gmail.com (Jonathan Monette) Date: Thu, 31 Mar 2011 21:04:12 -0500 Subject: [Swift-user] gensites documentation In-Reply-To: <154345204.53538.1301623294720.JavaMail.root@zimbra.anl.gov> References: <154345204.53538.1301623294720.JavaMail.root@zimbra.anl.gov> Message-ID: the -l option seems to be working in 0.92 which is what I am using right now. I am using 0.92 since my workflows seem to be getting deeper into them before they fail. On Thu, Mar 31, 2011 at 9:01 PM, Michael Wilde wrote: > > > ----- Original Message ----- > > Where is the gensites scripts documented? > > https://sites.google.com/site/swiftguide/home/managingsites > > >I want to use it to generate > > a pads-coaster implementation that I have written by hand to add into > > my run script but not sure how to use this. > > See if you can figure out how to proceed from the doc above and the source > code, and suggest what you think is a reasonable approach on swift-devel. > > We're discussing a few changes to gensites but want to move it forward to > production and supported use. > > I noted somewhere that the -l option mentioned doesnt seem to be in the > current code - not sure what the issue is. > > You can probably add new templates without much discussion. Ive suggested > some kind of help test mechanism for each template and its parameters. > > - Mike > > > > > -- > > Any intelligent fool can make things bigger and more complex... It > > takes a touch of genius - and a lot of courage to move in the opposite > > direction. > > - Albert Einstein > > > > > > > > _______________________________________________ > > Swift-user mailing list > > Swift-user at ci.uchicago.edu > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > -- Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. - Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From wilde at mcs.anl.gov Thu Mar 31 21:48:31 2011 From: wilde at mcs.anl.gov (Michael Wilde) Date: Thu, 31 Mar 2011 21:48:31 -0500 (CDT) Subject: [Swift-user] Problem in 0.92 source branch - do not use till corrected In-Reply-To: <1310504872.53534.1301623026989.JavaMail.root@zimbra.anl.gov> Message-ID: <1665932704.53596.1301626111997.JavaMail.root@zimbra.anl.gov> Dear Swift Users, On March 29 we discovered that the Release 0.92 branches of the Swift and CoG trees were changed after the release and a concurrency bug was introduced. If you are running Swift from this *source code* base, please revert back to a known-working release such as the 0.92 binary release if at all possible. We're working on restoring the 0.92 SVN branch to the correct state and will report back to this email list when that is done. - Mike -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory From tianyu491433909 at 163.com Thu Mar 31 22:47:50 2011 From: tianyu491433909 at 163.com (tianyu491433909) Date: Fri, 1 Apr 2011 11:47:50 +0800 (CST) Subject: [Swift-user] how to stop swift in the code Message-ID: How to stop swift in the code? -------------- next part -------------- An HTML attachment was scrubbed... URL: