From wilde at ci.uchicago.edu Sun Jun 1 23:36:36 2014 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Sun, 1 Jun 2014 23:36:36 -0500 (CDT) Subject: [Swift-commit] r7895 - in SwiftApps/tryswift: . scripts Message-ID: <20140602043636.507A99D7B2@svn.ci.uchicago.edu> Author: wilde Date: 2014-06-01 23:36:36 -0500 (Sun, 01 Jun 2014) New Revision: 7895 Added: SwiftApps/tryswift/hosts Modified: SwiftApps/tryswift/scripts/001-introduction.html SwiftApps/tryswift/scripts/001-introduction.swift Log: Edit intro swift and html. Add hosts file for all-host push (to be developed) Added: SwiftApps/tryswift/hosts =================================================================== --- SwiftApps/tryswift/hosts (rev 0) +++ SwiftApps/tryswift/hosts 2014-06-02 04:36:36 UTC (rev 7895) @@ -0,0 +1,14 @@ +swiftvm1.ci.uchicago.edu +swiftvm2.ci.uchicago.edu +swiftvm3.ci.uchicago.edu +swiftvm4.ci.uchicago.edu +ec2-23-20-13-105.compute-1.amazonaws.com +ec2-54-87-184-8.compute-1.amazonaws.com +ec2-54-82-124-154.compute-1.amazonaws.com +ec2-54-221-180-8.compute-1.amazonaws.com +ec2-54-82-219-10.compute-1.amazonaws.com +ec2-54-221-179-105.compute-1.amazonaws.com +ec2-54-82-146-148.compute-1.amazonaws.com +ec2-54-82-18-151.compute-1.amazonaws.com +ec2-54-87-83-42.compute-1.amazonaws.com +ec2-54-237-161-160.compute-1.amazonaws.com Modified: SwiftApps/tryswift/scripts/001-introduction.html =================================================================== --- SwiftApps/tryswift/scripts/001-introduction.html 2014-05-31 20:14:47 UTC (rev 7894) +++ SwiftApps/tryswift/scripts/001-introduction.html 2014-06-02 04:36:36 UTC (rev 7895) @@ -6,14 +6,33 @@

Welcome to Try Swift!

- If you are an existing Swift user, and would like to try testing your code, - please enter your code in the editor on the right and click "execute". +

+        This web interface lets you learn Swift from your browser - nothing to install.
+
+        You can run the built-in example scripts, change them, and run your own scripts.
+
+        - select a code example from the drop-down "Example" menu above
+        - click [Explain] to view an explanation of the  example
+        - click [Execute] (below) to run the example
+        - click [File outputs] to view the output files produced by your script
+        - click [Reset] to restore the example to its initial state (LOOSES YOUR CHANGES!)
+
+        The example scripts are:
+
+            Hello World   - Shows the basic syntax for running an app to produce a file
+            Foreach       - Introduces the foreach statement to run multiple apps in parallel
+            Multiple apps - Show dependencies between apps to specify a workflow
+            Multi-stage   - Example of a larger multi-stage workflow
+
+        To test your own script, enter some Swift code below, or in any other window
+
+        TrySwift gives you a few built-in "apps" to run, as needed by each example.
+        It executes on a pool of virtual machines, one VM per Swift run (at the moment).
+
+        The Swift language is explained in the Swift User Guide - see http://swift-lang.org/docs
+

-

- If you would like to learn Swift, select an example script above to learn more. -

- Modified: SwiftApps/tryswift/scripts/001-introduction.swift =================================================================== --- SwiftApps/tryswift/scripts/001-introduction.swift 2014-05-31 20:14:47 UTC (rev 7894) +++ SwiftApps/tryswift/scripts/001-introduction.swift 2014-06-02 04:36:36 UTC (rev 7895) @@ -1,26 +1,27 @@ /* + This web interface lets you learn Swift from your browser - nothing to install. - This "Try Swift" interface lets you learn Swift from your browser - nothing to install. - You can run the built-in example scripts, change them, and run your own scripts. - TrySwift gives you a few built-in "apps" to run, and executes on a set of virtual machines. + - select a code example from the drop-down "Example" menu above + - click [Explain] to view an explanation of the example + - click [Execute] (below) to run the example + - click [File outputs] to view the output files produced by your script + - click [Reset] to restore the example to its initial state (LOOSES YOUR CHANGES!) - Here's how: + The example scripts are: - - select the examples from the drop-down menu - - click [Explain] to read some background info on teh example - - click [Run] to run your code - - + Hello World - Shows the basic syntax for running an app to produce a file + Foreach - Introduces the foreach statement to run multiple apps in parallel + Multiple apps - Show dependencies between apps to specify a workflow + Multi-stage - Example of a larger multi-stage workflow - There are 4 other example scripts you may choose from: + To test your own script, enter some Swift code below, or in any other window - Hello World - Gives the basic syntax for running an application - Foreach - Introduces the foreach statement to run multiple apps at once - Multiple apps - Show dependencies between multiple applications - Multi-stage - Example of a larger multi-stage workflow + TrySwift gives you a few built-in "apps" to run, as needed by each example. + It executes on a pool of virtual machines, one VM per Swift run (at the moment). - To run your own script, enter the code below, or in any other Example window - + The Swift language is explained in the Swift User Guide - see http://swift-lang.org/docs */ +tracef("Hello, World! This is my %s code!\n","Swift"); From hategan at ci.uchicago.edu Mon Jun 2 00:16:38 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Mon, 2 Jun 2014 00:16:38 -0500 (CDT) Subject: [Swift-commit] r7896 - branches/release-0.95/src/org/griphyn/vdl/mapping/file Message-ID: <20140602051638.996239D7B2@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-02 00:16:38 -0500 (Mon, 02 Jun 2014) New Revision: 7896 Modified: branches/release-0.95/src/org/griphyn/vdl/mapping/file/StructuredRegularExpressionMapper.java Log: fixed broken merge Modified: branches/release-0.95/src/org/griphyn/vdl/mapping/file/StructuredRegularExpressionMapper.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/mapping/file/StructuredRegularExpressionMapper.java 2014-06-02 04:36:36 UTC (rev 7895) +++ branches/release-0.95/src/org/griphyn/vdl/mapping/file/StructuredRegularExpressionMapper.java 2014-06-02 05:16:38 UTC (rev 7896) @@ -70,7 +70,7 @@ Collection output = new ArrayList(); Collection sourceFields; try { - sourceFields = sourceHandle.getAllFields(); + sourceFields = sourceHandle.getFields(Path.CHILDREN); } catch (InvalidPathException ipe) { return Collections.emptyList(); From wilde at ci.uchicago.edu Mon Jun 2 00:23:19 2014 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Mon, 2 Jun 2014 00:23:19 -0500 (CDT) Subject: [Swift-commit] r7897 - www Message-ID: <20140602052319.696F59D7B2@svn.ci.uchicago.edu> Author: wilde Date: 2014-06-02 00:23:18 -0500 (Mon, 02 Jun 2014) New Revision: 7897 Modified: www/push_to.sh Log: Added new files to push_to.sh Modified: www/push_to.sh =================================================================== --- www/push_to.sh 2014-06-02 05:16:38 UTC (rev 7896) +++ www/push_to.sh 2014-06-02 05:23:18 UTC (rev 7897) @@ -73,6 +73,7 @@ images/button_bg_175.png images/bullet_next_32.png about/index.php +Swift-T/index.html index.html shCoreu.js js/jquery-1.5.1.min.js @@ -157,6 +158,7 @@ papers/Kickstarting2006.pdf papers/VDS-CMS.pdf papers/VirtualDataInCMS.pdf +papers/2014_HPDC14_GeMTC.pdf template_info/template.php usage/charts.swf usage/usage.php From wilde at ci.uchicago.edu Mon Jun 2 00:24:50 2014 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Mon, 2 Jun 2014 00:24:50 -0500 (CDT) Subject: [Swift-commit] r7898 - SwiftApps/tryswift Message-ID: <20140602052450.4BCC09D7B2@svn.ci.uchicago.edu> Author: wilde Date: 2014-06-02 00:24:49 -0500 (Mon, 02 Jun 2014) New Revision: 7898 Added: SwiftApps/tryswift/pushToAll.sh Log: Add script to push tryswift changes to all tryswift VM servers. Added: SwiftApps/tryswift/pushToAll.sh =================================================================== --- SwiftApps/tryswift/pushToAll.sh (rev 0) +++ SwiftApps/tryswift/pushToAll.sh 2014-06-02 05:24:49 UTC (rev 7898) @@ -0,0 +1,4 @@ +#! /bin/sh +for h in $(cat hosts); do + ssh $h "cd tryswift.svn; svn up; ./pushTo.sh /home/tryswift/tryswift" +done Property changes on: SwiftApps/tryswift/pushToAll.sh ___________________________________________________________________ Added: svn:executable + * From davidk at ci.uchicago.edu Mon Jun 2 16:03:42 2014 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 2 Jun 2014 16:03:42 -0500 (CDT) Subject: [Swift-commit] r7899 - tags Message-ID: <20140602210342.466289DA5B@svn.ci.uchicago.edu> Author: davidk Date: 2014-06-02 16:03:42 -0500 (Mon, 02 Jun 2014) New Revision: 7899 Added: tags/swift-0.95-RC6/ Log: 0.95 RC6 From davidk at ci.uchicago.edu Mon Jun 2 16:11:10 2014 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 2 Jun 2014 16:11:10 -0500 (CDT) Subject: [Swift-commit] r7900 - tags/swift-0.95-RC6/libexec Message-ID: <20140602211110.CCD049DA5B@svn.ci.uchicago.edu> Author: davidk Date: 2014-06-02 16:11:10 -0500 (Mon, 02 Jun 2014) New Revision: 7900 Modified: tags/swift-0.95-RC6/libexec/release.txt Log: Name release Modified: tags/swift-0.95-RC6/libexec/release.txt =================================================================== --- tags/swift-0.95-RC6/libexec/release.txt 2014-06-02 21:03:42 UTC (rev 7899) +++ tags/swift-0.95-RC6/libexec/release.txt 2014-06-02 21:11:10 UTC (rev 7900) @@ -1 +1 @@ -0.95 Branch SVN +0.95 RC6 From davidk at ci.uchicago.edu Mon Jun 2 16:57:24 2014 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Mon, 2 Jun 2014 16:57:24 -0500 (CDT) Subject: [Swift-commit] r7901 - SwiftApps/tryswift Message-ID: <20140602215724.363B39D047@svn.ci.uchicago.edu> Author: davidk Date: 2014-06-02 16:57:24 -0500 (Mon, 02 Jun 2014) New Revision: 7901 Modified: SwiftApps/tryswift/index.php Log: analytics Modified: SwiftApps/tryswift/index.php =================================================================== --- SwiftApps/tryswift/index.php 2014-06-02 21:11:10 UTC (rev 7900) +++ SwiftApps/tryswift/index.php 2014-06-02 21:57:24 UTC (rev 7901) @@ -4,6 +4,19 @@ Try Swift + + + From wilde at ci.uchicago.edu Mon Jun 2 18:50:17 2014 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Mon, 2 Jun 2014 18:50:17 -0500 (CDT) Subject: [Swift-commit] r7902 - in www: inc main support Message-ID: <20140602235017.6C04E9DA5B@svn.ci.uchicago.edu> Author: wilde Date: 2014-06-02 18:50:17 -0500 (Mon, 02 Jun 2014) New Revision: 7902 Modified: www/inc/footer2.php www/main/index.php www/support/index.php Log: Updated Support page. Added info email link to main page. Corrected Download button version number. Modified: www/inc/footer2.php =================================================================== --- www/inc/footer2.php 2014-06-02 21:57:24 UTC (rev 7901) +++ www/inc/footer2.php 2014-06-02 23:50:17 UTC (rev 7902) @@ -7,5 +7,5 @@

  • support
  • about
    • -
    The Swift project is supported by the National Science Foundation with additional support from NIH, Argonne National Laboratory and the University of Chicago Computation Institute.
    +
    The Swift project is supported by the National Science Foundation and US Department of Energy Office of Science, with additional support from the National Institutes of Health, Argonne National Laboratory and the University of Chicago Computation Institute.
    Modified: www/main/index.php =================================================================== --- www/main/index.php 2014-06-02 21:57:24 UTC (rev 7901) +++ www/main/index.php 2014-06-02 23:50:17 UTC (rev 7902) @@ -59,12 +59,14 @@ +
    + Join the Swift community or contact us at info at swift-lang.org
    Try Swift online
    Try Swift code examples right from your browser.
    Get Started
    Try the tutorial and start using Swift today!
    -
    Download Now
    0.95 current version
    2013/09/30 +
    Download Now
    0.94.1 current version
    2013/09/30
    -

    Support

    +
    -

    Reporting Problems

    - The best place to report problems and receive assistance is through the Swift mailing lists. - When reporting an issue, please be sure to include the Swift log files that were generated - as a part of your run. If the log files are too large to include in an email, please provide - access to the logs in a publicly readable directory or website. More information on these - mailing lists are provided below. -


    -

    Mailing Lists

    + +

    Contact the Swift team for more information or to report problems

    + +

    For more info about Swift, send email to the Swift team at info at swift-lang.org

    + +

    To report problems with Swift, send us a support ticket by emailing support at swift-lang.org


    + +

    Join the Swift Community via our email lists

    +

    Better yet, if you're using Swift or planning to try it, subscribe to the swift-user email list to hear what other users are doing, ask questions, and listen in on and join the community discussion.

    +

    If you'd like to contribute as a developer, or listen to the discussion of implementation issues, please join the swift-devel email list.

    +

    Note that you need to subscribe to these lists in order to send messages to them. Other messages will be moderated and may be significantly delayed.

    @@ -45,13 +48,13 @@ swift-user @@ -59,7 +62,7 @@ swift-devel +
    List - list page + subscribe archives - Swift user support and general questions. + Swift user community discussion and general questions.
    - list page + subscribe archives @@ -69,6 +72,7 @@ The Swift developers mailing list.
    +

     

    -

     

    + + +

    Reporting Problems

    +

    The best way to report problems and receive assistance from the Swift team and community members is to join the swift-user email list.

    +

    When reporting an issue to swift-support, please be very precise and detailed. Send us all relevant files and output, and be sure to include the Swift *.log files that were generated + as a part of your run. If the log files are too large to include in an email, please provide + access to the logs at a publicly readable URL or directory.

    +
    +

    Search Mailing List Archive

    Loading
    From wilde at ci.uchicago.edu Mon Jun 2 23:37:24 2014 From: wilde at ci.uchicago.edu (wilde at ci.uchicago.edu) Date: Mon, 2 Jun 2014 23:37:24 -0500 (CDT) Subject: [Swift-commit] r7903 - in www: . main Message-ID: <20140603043724.207119D047@svn.ci.uchicago.edu> Author: wilde Date: 2014-06-02 23:37:24 -0500 (Mon, 02 Jun 2014) New Revision: 7903 Added: www/push_to_all.sh Modified: www/main/index.php Log: Add push_to_all.sh (prelim version from David). Revert to direct link for Swift-T info, till we fix the blank-page problem with that link. Modified: www/main/index.php =================================================================== --- www/main/index.php 2014-06-02 23:50:17 UTC (rev 7902) +++ www/main/index.php 2014-06-03 04:37:24 UTC (rev 7903) @@ -176,7 +176,8 @@
    Next Generation Swift

    Swift/T runs millions of tasks per second on supercomputers.

    -

    Learn more...

    + +

    Learn more...

    Added: www/push_to_all.sh =================================================================== --- www/push_to_all.sh (rev 0) +++ www/push_to_all.sh 2014-06-03 04:37:24 UTC (rev 7903) @@ -0,0 +1,7 @@ +hosts="ec2-23-20-156-199.compute-1.amazonaws.com ec2-54-87-91-46.compute-1.amazonaws.com ec2-54-196-24-118.compute-1.amazonaws.com ec2-23-20-140-94.compute-1.amazonaws.com ec2-54-82-19-125.compute-1.amazonaws.com ec2-184-72-180-226.compute-1.amazonaws.com ec2-23-20-187-161.compute-1.amazonaws.com ec2-54-81-204-169.compute-1.amazonaws.com ec2-54-205-193-6.compute-1.amazonaws.com ec2-54-237-183-152.compute-1.amazonaws.com" + +for host in $hosts +do + scp www.tar.gz ubuntu@$host: + ssh ubuntu@$host "cd /var/www && tar xfz /home/ubuntu/www.tar.gz" +done Property changes on: www/push_to_all.sh ___________________________________________________________________ Added: svn:executable + * From hategan at ci.uchicago.edu Tue Jun 3 15:38:32 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 3 Jun 2014 15:38:32 -0500 (CDT) Subject: [Swift-commit] r7904 - trunk/src/org/griphyn/vdl/karajan/lib Message-ID: <20140603203832.B5347178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-03 15:38:32 -0500 (Tue, 03 Jun 2014) New Revision: 7904 Modified: trunk/src/org/griphyn/vdl/karajan/lib/GetStagingInfo.java Log: allow duplicate files in stage-ins Modified: trunk/src/org/griphyn/vdl/karajan/lib/GetStagingInfo.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/lib/GetStagingInfo.java 2014-06-03 04:37:24 UTC (rev 7903) +++ trunk/src/org/griphyn/vdl/karajan/lib/GetStagingInfo.java 2014-06-03 20:38:32 UTC (rev 7904) @@ -54,9 +54,9 @@ private static class Info { Set remoteDirNames = Collections.emptySet(); - List inFiles = Collections.emptyList(); - List outFiles = Collections.emptyList(); - List collectPatterns = Collections.emptyList(); + Set inFiles = Collections.emptySet(); + Set outFiles = Collections.emptySet(); + Set collectPatterns = Collections.emptySet(); } @Override @@ -74,15 +74,13 @@ catch (HandleOpenException e) { throw new ExecutionException(e.getMessage(), e); } - ret.add(info.remoteDirNames); - ret.add(info.inFiles); - ret.add(info.outFiles); - ret.add(info.collectPatterns); + ret.add(new ArrayList(info.remoteDirNames)); + ret.add(new ArrayList(info.inFiles)); + ret.add(new ArrayList(info.outFiles)); + ret.add(new ArrayList(info.collectPatterns)); return null; } - - private void addPaths(Info info, Collection vars, boolean out) throws HandleOpenException { for (DSHandle var : vars) { if (!var.getType().hasMappedComponents()) { @@ -118,7 +116,7 @@ } - private List addOne(AbsFile f, Info info, List files) { + private Set addOne(AbsFile f, Info info, Set files) { String dir = f.getDirectory(); if (dir != null) { if (info.remoteDirNames.isEmpty()) { @@ -127,13 +125,12 @@ info.remoteDirNames.add(remoteDir(f, dir)); } if (files.isEmpty()) { - files = new ArrayList(); + files = new HashSet(); } files.add(f); return files; } - private Mapper getMapper(DSHandle var) { Mapper m = var.getMapper(); if (m == null) { @@ -142,7 +139,6 @@ return m; } - private String remoteDir(AbsFile f, String dir) { if ("file".equals(f.getProtocol())) { return PathUtils.remotePathName(dir); From hategan at ci.uchicago.edu Tue Jun 3 15:40:12 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 3 Jun 2014 15:40:12 -0500 (CDT) Subject: [Swift-commit] r7905 - trunk/tests/language-behaviour/IO Message-ID: <20140603204012.CE067178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-03 15:40:12 -0500 (Tue, 03 Jun 2014) New Revision: 7905 Added: trunk/tests/language-behaviour/IO/170-duplicate-input.in trunk/tests/language-behaviour/IO/170-duplicate-input.swift Log: added test for duplicate stageins Added: trunk/tests/language-behaviour/IO/170-duplicate-input.in =================================================================== --- trunk/tests/language-behaviour/IO/170-duplicate-input.in (rev 0) +++ trunk/tests/language-behaviour/IO/170-duplicate-input.in 2014-06-03 20:40:12 UTC (rev 7905) @@ -0,0 +1 @@ +Bla Added: trunk/tests/language-behaviour/IO/170-duplicate-input.swift =================================================================== --- trunk/tests/language-behaviour/IO/170-duplicate-input.swift (rev 0) +++ trunk/tests/language-behaviour/IO/170-duplicate-input.swift 2014-06-03 20:40:12 UTC (rev 7905) @@ -0,0 +1,13 @@ +type file; + + +file input <"170-duplicate-input.in">; + +file output <"170-duplicate-input.out">; + +app (file t) cat(file m, file n) { + cat @filename(m) @filename(n) stdout=@filename(t); +} + +output = cat(input, input); + From swift at ci.uchicago.edu Fri Jun 6 15:11:20 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 6 Jun 2014 15:11:20 -0500 (CDT) Subject: [Swift-commit] cog r3909 Message-ID: <20140606201120.DDED88D000F4@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3909 | yadunand | 2014-06-06 15:07:40 -0500 (Fri, 06 Jun 2014) | 3 lines Committing latest for Tim ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/tests/coasters/start_active_coaster.sh =================================================================== --- modules/provider-coaster-c-client/tests/coasters/start_active_coaster.sh (revision 0) +++ modules/provider-coaster-c-client/tests/coasters/start_active_coaster.sh (revision 3909) @@ -0,0 +1,8 @@ +#!/bin/bash + +COASTER_PORT=53001 +export COG_OPTS=-Duser.home=/scratch/midway/yadunand/swiftwork +echo "Starting active coasters" +echo "Using swift-version : $(swift -version)" + +coaster-service -p $COASTER_PORT -nosec | tee coaster_logs.txt Property changes on: modules/provider-coaster-c-client/tests/coasters/start_active_coaster.sh ___________________________________________________________________ Added: svn:executable + * Index: modules/provider-coaster-c-client/tests/coasters/sites.xml =================================================================== --- modules/provider-coaster-c-client/tests/coasters/sites.xml (revision 3908) +++ modules/provider-coaster-c-client/tests/coasters/sites.xml (working copy) @@ -1,12 +1,12 @@ - - passive + + active 4 100 10000 - /sandbox/yadunand + /scratch/midway/yadunand/swiftwork From yadunandb at ci.uchicago.edu Mon Jun 9 12:03:05 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 9 Jun 2014 12:03:05 -0500 (CDT) Subject: [Swift-commit] r7906 - in branches/release-0.95/tests: . groups multi_remote sites/uc3 stress stress/IO/beagle stress/IO/multiple stress/IO/uc3 stress/apps/modis_uc3 stress/internals stress/jobs_per_node stress/random_fail Message-ID: <20140609170305.EFAE09D053@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-09 12:03:04 -0500 (Mon, 09 Jun 2014) New Revision: 7906 Added: branches/release-0.95/tests/stress/jobs_per_node/ branches/release-0.95/tests/stress/jobs_per_node/combiner.sh branches/release-0.95/tests/stress/jobs_per_node/count_jobs.sh branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.args branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.check.sh branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.setup.sh branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.stdout branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.swift branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.timeout branches/release-0.95/tests/stress/jobs_per_node/run branches/release-0.95/tests/stress/jobs_per_node/sites.template.xml branches/release-0.95/tests/stress/jobs_per_node/swift.log branches/release-0.95/tests/stress/jobs_per_node/swift.properties branches/release-0.95/tests/stress/jobs_per_node/tc.data branches/release-0.95/tests/stress/jobs_per_node/teragen_wrap.sh branches/release-0.95/tests/stress/jobs_per_node/title.txt branches/release-0.95/tests/stress/random_fail/ branches/release-0.95/tests/stress/random_fail/Bug_info branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.check.sh branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.setup.sh branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.swift branches/release-0.95/tests/stress/random_fail/run branches/release-0.95/tests/stress/random_fail/sites.template.xml branches/release-0.95/tests/stress/random_fail/swift.properties branches/release-0.95/tests/stress/random_fail/tc.template.data branches/release-0.95/tests/stress/random_fail/title.txt Modified: branches/release-0.95/tests/groups/group-daily-remote.sh branches/release-0.95/tests/multi_remote/remote_driver.check.sh branches/release-0.95/tests/multi_remote/remote_driver.setup.sh branches/release-0.95/tests/sites/uc3/sites.template.xml branches/release-0.95/tests/stress/IO/beagle/sites.template.xml branches/release-0.95/tests/stress/IO/multiple/sites.template.xml branches/release-0.95/tests/stress/IO/uc3/sites.template.xml branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml branches/release-0.95/tests/stress/internals/tc.template.data branches/release-0.95/tests/user_test.sh branches/release-0.95/tests/user_yadu.sh Log: Updated sites.xml for UC3. Several old updates Modified: branches/release-0.95/tests/groups/group-daily-remote.sh =================================================================== --- branches/release-0.95/tests/groups/group-daily-remote.sh 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/groups/group-daily-remote.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -33,7 +33,7 @@ $TESTDIR/sites/mac-frisbee \ $TESTDIR/sites/blues \ $TESTDIR/sites/fusion \ - $TESTDIR/sites/raven \ + #$TESTDIR/sites/raven \ # Remote-cluster IO tests $TESTDIR/stress/IO/beagle \ @@ -51,8 +51,6 @@ $TESTDIR/stress/apps/modis_uc3 \ # $TESTDIR/stress/apps/modis_multiple\ - # Local stress tests - $TESTDIR/stress/internals \ # Local cluster tests. $TESTDIR/stress/local_cluster \ $TESTDIR/stress/random_fail \ Modified: branches/release-0.95/tests/multi_remote/remote_driver.check.sh =================================================================== --- branches/release-0.95/tests/multi_remote/remote_driver.check.sh 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/multi_remote/remote_driver.check.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -15,8 +15,8 @@ SITES_PASSED=`ls out/*out | wc -l` HOME=$PWD MAIL="$HOME/Mail.content" -#VERSION=`grep "^Swift.*swift-.*cog.*" remote_driver.stdout | cut -d " " -f -2 | sed 's/\ /-/g' | tr '[A-Z]' '[a-z]'` -VERSION="swift-trunk" +VERSION=`grep "^Swift.*swift-.*cog.*" remote_driver.stdout | cut -d " " -f -2 | sed 's/\ /-/g' | tr '[A-Z]' '[a-z]'` +#VERSION="swift-trunk" cd out; echo "Test-Results" > $MAIL Modified: branches/release-0.95/tests/multi_remote/remote_driver.setup.sh =================================================================== --- branches/release-0.95/tests/multi_remote/remote_driver.setup.sh 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/multi_remote/remote_driver.setup.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -1,8 +1,8 @@ #!/bin/bash -[ ! -z $COG_URL ] || COG_URL=https://svn.code.sf.net/p/cogkit/svn/trunk/src/cog -[ ! -z $SWIFT_URL ] || SWIFT_URL=https://svn.ci.uchicago.edu/svn/vdl2/trunk -[ ! -z $SWIFT_VERSION ] || SWIFT_VERSION=trunk +[ ! -z $COG_URL ] || COG_URL=https://svn.code.sf.net/p/cogkit/svn/branches/4.1.11/src/cog +[ ! -z $SWIFT_URL ] || SWIFT_URL=https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.95 +[ ! -z $SWIFT_VERSION ] || SWIFT_VERSION=0.95 [ ! -z $BEAGLE_USERNAME ] || BEAGLE_USERNAME="yadunandb" [ ! -z $MIDWAY_USERNAME ] || MIDWAY_USERNAME="yadunand" [ ! -z $UC3_USERNAME ] || UC3_USERNAME="yadunand" Modified: branches/release-0.95/tests/sites/uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/uc3/sites.template.xml 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/sites/uc3/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -2,7 +2,7 @@ - + 10.00 10000 1 @@ -13,7 +13,7 @@ 1 1 1 - nonshared - . + "Swift" + /tmp/swiftwork - \ No newline at end of file + Modified: branches/release-0.95/tests/stress/IO/beagle/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/IO/beagle/sites.template.xml 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/stress/IO/beagle/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -8,10 +8,10 @@ pbs.aprun;pbs.mpp;depth=24 3700 00:25:00 - development + /lustre/beagle/{env.USER}/swiftwork - 20 - 10 + 2 + 4 1 4.80 10000 @@ -19,4 +19,4 @@ /tmp/{env.USER}/swiftwork - \ No newline at end of file + Modified: branches/release-0.95/tests/stress/IO/multiple/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/IO/multiple/sites.template.xml 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/stress/IO/multiple/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -1,5 +1,5 @@ - - + + 10.00 @@ -24,7 +24,6 @@ 100 100 - pbs.aprun;pbs.mpp;depth=24;pbs.resource_list=advres=wilde.1768 3600 00:05:00 /lustre/beagle/{beagle.USER}/swiftwork Modified: branches/release-0.95/tests/stress/IO/uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/IO/uc3/sites.template.xml 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/stress/IO/uc3/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -1,17 +1,21 @@ - + + + 10.00 10000 1 3600 - 00:30:00 + 00:10:00 100 100 - 1000 + 2 1 1 nonshared + "Swift" . - \ No newline at end of file + + Modified: branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -14,6 +14,7 @@ 1 1 "group_friends.{env.USER}" + "Swift" nonshared . Modified: branches/release-0.95/tests/stress/internals/tc.template.data =================================================================== --- branches/release-0.95/tests/stress/internals/tc.template.data 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/stress/internals/tc.template.data 2014-06-09 17:03:04 UTC (rev 7906) @@ -1,2 +1,2 @@ -local bash /bin/bash null null null -local sleep /bin/sleep null null null +localhost bash /bin/bash null null null +localhost sleep /bin/sleep null null null Added: branches/release-0.95/tests/stress/jobs_per_node/combiner.sh =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/combiner.sh (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/combiner.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,16 @@ +#!/bin/bash + +FILES=$* +SUM=0 +COUNT=0 + +for file in $* +do + RES=($(awk '{ sum += $1 } END { print sum,NR }' $file)) + echo "${RES[0]} ${RES[1]}" + SUM=$(($SUM+${RES[0]})) + COUNT=$(($COUNT+${RES[1]})) +done +echo "SUM : $SUM" +echo "COUNT: $COUNT" +exit 0 Added: branches/release-0.95/tests/stress/jobs_per_node/count_jobs.sh =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/count_jobs.sh (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/count_jobs.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,16 @@ +#!/bin/bash + +SELF="count_jobs.sh" +SLEEPTIME=60 + + +BEFORE=$(($RANDOM%$SLEEPTIME)) +AFTER=$(($SLEEPTIME-$BEFORE)) + +sleep $BEFORE +echo "NODE $(hostname -f)" +ACTIVE=`ps -u $USER | grep $SELF | wc -l` +echo "ps -u $USER | grep | wc -l" +echo "ACTIVE $ACTIVE" +echo "SPLIT $BEFORE:$AFTER" +sleep $AFTER \ No newline at end of file Property changes on: branches/release-0.95/tests/stress/jobs_per_node/count_jobs.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.args =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.args (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.args 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1 @@ +-loops=10 Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.check.sh =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.check.sh (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.check.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,22 @@ +#!/bin/bash + +BASE=${0%.check.sh} +ARGS=`cat $BASE.args | sed 's/-loops=//'` + +exit 0 +EXPECTED=$(($ARGS * 10000)) + +if [ -f "final_result" ];then + RESULT=($(tail -n 1 final_result)) + echo "RESULT line : ${RESULT[*]}" + echo "EXPECTED = $EXPECTED" + echo "ACTUAL = ${RESULT[1]}" +fi + +if [[ "${RESULT[1]}" == "$EXPECTED" ]] +then + echo "Result matched" +else + echo "Result does not match expectation" >&2 + exit 1 +fi Property changes on: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.check.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.setup.sh =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.setup.sh (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.setup.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,100 @@ +#!/bin/bash + +jobspernode=$(( $RANDOM % 7 + 1 )) +LOOPS=$(( $(($RANDOM%4 + 1)) * 2 )) + +HOST=$(hostname -f) + +ARGS_FILE=${0%.setup.sh}.args + +if [[ "$HOST" == *midway* ]]; then + echo "On Midway" + echo "midway bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *beagle* ]]; then + echo "On Beagle" + echo "beagle bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *mcs* ]]; then + echo "On MCS" + echo "mcs bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *uc3* ]]; then + echo "On UC3" + echo "uc3 bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == blogin*lcrc* ]]; then + echo "On Blues at LCRC " + echo "blues bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == flogin*lcrc* ]]; then + echo "On Fusion at LCRC " + echo "fusion bash /bin/bash null null null" > tc.data +else + echo "On unidentified machine, using defaults" + echo "local bash /bin/bash null null null" > tc.data +fi + +echo "-loops=$(( $jobspernode * $LOOPS ))" > $ARGS_FILE +cat sites.xml | sed "s/JOBSPERNODE/$jobspernode/" > tmp && mv tmp sites.xml + +cp sites.xml sites.xml.before +if [[ -z $MIDWAY_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/MIDWAY_USERNAME/$MIDWAY_USERNAME/" > tmp && mv tmp\ + sites.xml +fi +if [[ -z $UC3_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/UC3_USERNAME/$UC3_USERNAME/" > tmp && mv tmp si\ +tes.xml +fi +if [[ -z $BEAGLE_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/BEAGLE_USERNAME/$BEAGLE_USERNAME/" > tmp && mv \ +tmp sites.xml +fi +if [[ -z $MCS_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/MCS_USERNAME/$MCS_USERNAME/" > tmp && mv \ +tmp sites.xml +fi +if [[ -z $BLUES_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/BLUES_USERNAME/$BLUES_USERNAME/" > tmp && mv \ +tmp sites.xml +fi +if [[ -z $FUSION_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/FUSION_USERNAME/$FUSION_USERNAME/" > tmp && mv \ +tmp sites.xml +fi +cp sites.xml sites.xml.after + +cat<<'EOF' > count_jobs.sh +#!/bin/bash + +SELF="count_jobs.sh" +SLEEPTIME=60 + + +BEFORE=$(($RANDOM%$SLEEPTIME)) +AFTER=$(($SLEEPTIME-$BEFORE)) + +sleep $BEFORE +echo "NODE $(hostname -f)" +ACTIVE=`ps -u $USER | grep $SELF | wc -l` +ps -u $USER +echo "ps -u $USER | grep | wc -l" +echo "ACTIVE $ACTIVE" +echo "SPLIT $BEFORE:$AFTER" +sleep $AFTER +EOF + Property changes on: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.setup.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.stdout =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.stdout (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.stdout 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,3 @@ +Swift 0.94 swift-r6888 cog-r3762 + +RunID: 20130822-2202-c5ih1pj1 Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.swift =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.swift (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.swift 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,18 @@ +type file; +type script; + +app (file out, file err) app_run (script run) +{ + bash @run stdout=@out stderr=@err; +} + +file f_out[] ; +file f_err[] ; + +script wrapper <"count_jobs.sh">; + +int loop = @toInt(@arg("loops","10")); + +foreach item,i in [0:loop-1] { + (f_out[i], f_err[i]) = app_run(wrapper); +} Added: branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.timeout =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.timeout (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/jobs_per_node.timeout 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1 @@ +300 \ No newline at end of file Added: branches/release-0.95/tests/stress/jobs_per_node/run =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/run (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/run 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,27 @@ +#!/bin/bash + +#PATH=/scratch/midway/yadunand/swift-0.94RC2/cog/modules/swift/dist/swift-svn/bin:$PATH + +echo "Swift location: "; which swift +echo "Swift version : "; swift -version + +export MIDWAY_USERNAME=yadunand +export BEAGLE_USERNAME=yadunandb +export MCS_USERNAME=yadunand +export UC3_USERNAME=yadunand + +SCRIPT=jobs_per_node.swift +BASE=${SCRIPT%.swift} + +rm $BASE.stdout +cat title.txt + +cp sites.template.xml sites.xml +./$BASE.setup.sh + +ARGS=$(cat $BASE.args) +swift -tc.file tc.data -config swift.properties -sites.file sites.xml $BASE.swift ${ARGS[*]} | tee -a $BASE.stdout + +rm -rf *{swiftx,kml} $BASE-* _concurrent* failed* &> /dev/null + +./$BASE.check.sh \ No newline at end of file Property changes on: branches/release-0.95/tests/stress/jobs_per_node/run ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/jobs_per_node/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/sites.template.xml (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,128 @@ + + + + + JOBSPERNODE + 00:15:00 + 100 + 100 + 1 + 1 + 1 + 0.03 + 10000 + + . + + + + + JOBSPERNODE + 100 + 100 + + 36000 + 01:05:00 + /lustre/beagle/BEAGLE_USERNAME/swiftwork + 4 + 1 + 1 + 4.80 + 10000 + /tmp/BEAGLE_USERNAME/swiftwork + + + + + + 3600 + JOBSPERNODE + 8 + route + 2 + 4 + 4 + 2.20 + 10000 + 1000 + 1000 + /tmp/BLUES_USERNAME/swiftwork + + + + + + 3600 + JOBSPERNODE + 4 + 2 + 2 + shared + 5.99 + 10000 + 1000 + 1000 + /homes/FUSION_USERNAME/swiftwork + + + + + sandyb + JOBSPERNODE + 36000 + 01:05:00 + 100 + 100 + 4 + 1 + 1 + .64 + 10000 + /tmp/MIDWAY_USERNAME + + + + + + 10.00 + 10000 + JOBSPERNODE + 36000 + 01:05:00 + 100 + 100 + 1000 + 1 + 1 + + nonshared + /home/UC3_USERNAME/swiftwork + + + + + JOBSPERNODE + 100 + 100 + 36000 + 01:05:00 + 0.0799 + 10000 + /sandbox/MCS_USERNAME/ + /sandbox/MCS_USERNAME/swiftwork + + + + + + JOBSPERNODE + 100 + 100 + 3600 + 00:05:00 + 0.0799 + 10000 + /sandbox/MCS_USERNAME/swiftwork + + + Added: branches/release-0.95/tests/stress/jobs_per_node/swift.log =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/swift.log (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/swift.log 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,4 @@ +2013-08-22 22:01:41,282+0000 DEBUG Loader Swift started +2013-08-22 22:01:41,557+0000 DEBUG Loader Swift started +2013-08-22 22:02:03,942+0000 DEBUG Loader Swift started +2013-08-22 22:02:04,214+0000 DEBUG Loader Swift started Added: branches/release-0.95/tests/stress/jobs_per_node/swift.properties =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/swift.properties (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/swift.properties 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,8 @@ +use.provider.staging=true +use.wrapper.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +lazy.errors=false +provider.staging.pin.swiftfiles=false +sitedir.keep=true +tcp.port.range=50000,51000 \ No newline at end of file Added: branches/release-0.95/tests/stress/jobs_per_node/tc.data =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/tc.data (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/tc.data 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1 @@ +midway bash /bin/bash null null null Added: branches/release-0.95/tests/stress/jobs_per_node/teragen_wrap.sh =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/teragen_wrap.sh (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/teragen_wrap.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,15 @@ +#!/bin/bash + +# By default with ARG1:100 and SLICESIZE=10000, this script will generate +# 10^6 records. +ARG1=1 +[ ! -z $1 ] && ARG1=$1 + +FILE="input_$RANDOM.txt" +LOWERLIMIT=0 +UPPERLIMIT=1000000 # 10^9 +SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file +#SLICESIZE=1000 # 10^3 If padded to 100B would result + +shuf -i $LOWERLIMIT-$UPPERLIMIT -n $(($SLICESIZE*$ARG1)) | awk '{printf "%-99s\n", $0}' +exit 0 Added: branches/release-0.95/tests/stress/jobs_per_node/title.txt =================================================================== --- branches/release-0.95/tests/stress/jobs_per_node/title.txt (rev 0) +++ branches/release-0.95/tests/stress/jobs_per_node/title.txt 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,2 @@ +Test for the jobspernode swift config attribute + Added: branches/release-0.95/tests/stress/random_fail/Bug_info =================================================================== --- branches/release-0.95/tests/stress/random_fail/Bug_info (rev 0) +++ branches/release-0.95/tests/stress/random_fail/Bug_info 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,22 @@ +Regression tests for Bug 1067 +| Link -> http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1067 +Swift 0.94 swift-r6888 cog-r3762 + +Exception in sh: + Arguments: [randfail.sh, 50, 0] + Host: local + Directory: rand_fail_Bug1067-20130820-1749-6n7zbux4/jobs/6/sh-6abm13el + stderr.txt: Failing 11 < 50 + stdout.txt: +Caused by: Application /bin/bash failed with an exit code of 255 + +Exception in sh: + Arguments: [randfail.sh, 50, 0] + Host: local + Directory: rand_fail_Bug1067-20130820-1749-6n7zbux4/jobs/8/sh-8abm13el + stderr.txt: Failing 19 < 50 + stdout.txt: +Caused by: Application /bin/bash failed with an exit code of 255 + +Execution failed: + Got one name (derr) and 0 values: [] \ No newline at end of file Added: branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.check.sh =================================================================== --- branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.check.sh (rev 0) +++ branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.check.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,17 @@ +#!/bin/bash + +if [ ! -f ${0%.check.sh}.stdout ] +then + echo "${$0%.check.sh}.stdout missing" + exit -1 +fi + +grep "Got one name (derr)" ${0%.check.sh}.stdout +if [ "$?" == 0 ] +then + echo "EXIT : REGRESSION FOUND!" >&2 + exit -1 +else + echo "Test passed" +fi +exit 0 \ No newline at end of file Property changes on: branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.check.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.setup.sh =================================================================== --- branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.setup.sh (rev 0) +++ branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.setup.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,22 @@ +#!/bin/bash + +# Setup script will just output the following file + +cat<<'EOF' > randfail.sh +#!/bin/bash + +FAIL_PROBABILITY=$1 +DELAY=$2 + +ITEM=$(($RANDOM%100)) +sleep $2 + +if (( "$ITEM" <= "$FAIL_PROBABILITY" )) +then + echo "Failing $ITEM < $FAIL_PROBABILITY" >&2 + exit -1 +fi +echo "Not failing $ITEM > $FAIL_PROBABILITY" +exit 0 +EOF + Property changes on: branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.setup.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.swift =================================================================== --- branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.swift (rev 0) +++ branches/release-0.95/tests/stress/random_fail/rand_fail_Bug1067.swift 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,34 @@ +type file; + +file script<"randfail.sh">; + +app (file ofile1, file ofile2) quicklyFailingApp(file script, int failchance, +int delay) +{ + sh @script failchance delay stdout=@ofile1 stderr=@ofile2; +} + +app (file ofile) someApp3(file ifile, file jfile, file kfile) +{ + sh "-c" @strcat("cat ", at filename(ifile)) stdout=@ofile; +} + +app (file ofile) someApp(file ifile) +{ + sh "-c" @strcat("cat ", at filename(ifile)) stdout=@ofile; +} + +app sleep (int sec) +{ + sh "-c" @strcat("sleep ",sec); +} + +int sufficientlyLargeNumber = 100; + +file a[]; +foreach i in [0:sufficientlyLargeNumber] { + file f1; + file f2; + (f1,f2) = quicklyFailingApp(script,50,0); + a[i] = someApp(f2); +} Added: branches/release-0.95/tests/stress/random_fail/run =================================================================== --- branches/release-0.95/tests/stress/random_fail/run (rev 0) +++ branches/release-0.95/tests/stress/random_fail/run 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,21 @@ +#!/bin/bash + +PATH=/scratch/midway/yadunand/swift-0.94RC2/cog/modules/swift/dist/swift-svn/bin:$PATH + + +echo "Swift location: "; which swift +echo "Swift version : "; swift -version + +rm rand_fail_Bug1067.stdout +cat title.txt + +./rand_fail_Bug1067.setup.sh + +for i in `seq 1 10` +do +swift -tc.file tc.template.data -config swift.properties -sites.file sites.template.xml rand_fail_Bug1067.swift | tee -a rand_fail_Bug1067.stdout + +rm -rf *{swiftx,kml} rand_fail_Bug1067-* _concurrent* failed* +done + +./rand_fail_Bug1067.check.sh \ No newline at end of file Property changes on: branches/release-0.95/tests/stress/random_fail/run ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/random_fail/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/random_fail/sites.template.xml (rev 0) +++ branches/release-0.95/tests/stress/random_fail/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,20 @@ + + + + + + sandyb + 16 + 36000 + 00:10:00 + 100 + 100 + 4 + 1 + 1 + .64 + 10000 + /tmp/MIDWAY_USERNAME + + + Added: branches/release-0.95/tests/stress/random_fail/swift.properties =================================================================== --- branches/release-0.95/tests/stress/random_fail/swift.properties (rev 0) +++ branches/release-0.95/tests/stress/random_fail/swift.properties 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,11 @@ +wrapperlog.always.transfer=true +sitedir.keep=true +file.gc.enabled=false +status.mode=provider + +execution.retries=5 +lazy.errors=true + +use.wrapper.staging=false +use.provider.staging=false +provider.staging.pin.swiftfiles=falsewrapperlog.always.transfer=true Added: branches/release-0.95/tests/stress/random_fail/tc.template.data =================================================================== --- branches/release-0.95/tests/stress/random_fail/tc.template.data (rev 0) +++ branches/release-0.95/tests/stress/random_fail/tc.template.data 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,2 @@ +local perl /usr/bin/perl null null null +local sh /bin/bash Added: branches/release-0.95/tests/stress/random_fail/title.txt =================================================================== --- branches/release-0.95/tests/stress/random_fail/title.txt (rev 0) +++ branches/release-0.95/tests/stress/random_fail/title.txt 2014-06-09 17:03:04 UTC (rev 7906) @@ -0,0 +1,2 @@ +Regression tests for Bug 1067 +| Link -> http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1067 Modified: branches/release-0.95/tests/user_test.sh =================================================================== --- branches/release-0.95/tests/user_test.sh 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/user_test.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -26,4 +26,4 @@ then export GLOBUS_HOSTNAME=swift.rcc.uchicago.edu export GLOBUS_TCP_PORT_RANGE=50000,51000 -fi; \ No newline at end of file +fi; Modified: branches/release-0.95/tests/user_yadu.sh =================================================================== --- branches/release-0.95/tests/user_yadu.sh 2014-06-03 20:40:12 UTC (rev 7905) +++ branches/release-0.95/tests/user_yadu.sh 2014-06-09 17:03:04 UTC (rev 7906) @@ -7,12 +7,21 @@ export COMM_USERNAME="yadunandb" export BRID_USERNAME="yadunandb" export FROM_MAIL="Test-Engine at midway001" -export TO_MAIL="yadudoc1729 at gmail.com" +export TO_MAIL="yadudoc1729 at gmail.com davidkelly999 at uchicago.edu wilde at mcs.anl.gov" +#export TO_MAIL="yadudoc1729 at gmail.com" -#export REMOTE_DRIVER_FASTSETUP="false" -export REMOTE_DRIVER_FASTSETUP="true" -export SWIFT_TAR_FILE="/scratch/midway/yadunand/swift-trunk.tar" +################# MUST UPDATE FOR EACH REVISION ######################### +COG_URL=https://svn.code.sf.net/p/cogkit/svn/branches/4.1.11/src/cog +SWIFT_URLk=https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.95 +export SWIFT_TAR_FILE="" +export SWIFT_VERSION_OVERRIDE="swift-0.95" +export SWIFT_VERSION="swift-0.95" +################# MUST UPDATE FOR EACH REVISION ######################### +export REMOTE_DRIVER_FASTSETUP="false" +#export REMOTE_DRIVER_FASTSETUP="true" +export CLEAN_CHECKOUT="yes" + export KILL_JAVA="false" # Extra performance stats @@ -25,4 +34,4 @@ then export GLOBUS_HOSTNAME=swift.rcc.uchicago.edu export GLOBUS_TCP_PORT_RANGE=50000,51000 -fi; \ No newline at end of file +fi; From yadunandb at ci.uchicago.edu Mon Jun 9 14:52:23 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 9 Jun 2014 14:52:23 -0500 (CDT) Subject: [Swift-commit] r7907 - branches/release-0.95/tests/sites/uc3 Message-ID: <20140609195223.052219D053@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-09 14:52:22 -0500 (Mon, 09 Jun 2014) New Revision: 7907 Removed: branches/release-0.95/tests/sites/uc3/sanity.setup.sh branches/release-0.95/tests/sites/uc3/sanity.source.sh branches/release-0.95/tests/sites/uc3/sanity.swift branches/release-0.95/tests/sites/uc3/sanity.timeout Modified: branches/release-0.95/tests/sites/uc3/sites.template.xml Log: UC3 sanity tests, fixed. Deleted: branches/release-0.95/tests/sites/uc3/sanity.setup.sh =================================================================== --- branches/release-0.95/tests/sites/uc3/sanity.setup.sh 2014-06-09 17:03:04 UTC (rev 7906) +++ branches/release-0.95/tests/sites/uc3/sanity.setup.sh 2014-06-09 19:52:22 UTC (rev 7907) @@ -1,11 +0,0 @@ -#!/bin/bash - -USERNAME=$UC3_USERNAME - -if [[ -z $USERNAME ]] -then - echo "Remote username not provided. Skipping sites configs" -else - ls *xml - cat sites.xml | sed "s/{env.USER}/$USERNAME/" > tmp && mv tmp sites.xml -fi Deleted: branches/release-0.95/tests/sites/uc3/sanity.source.sh =================================================================== --- branches/release-0.95/tests/sites/uc3/sanity.source.sh 2014-06-09 17:03:04 UTC (rev 7906) +++ branches/release-0.95/tests/sites/uc3/sanity.source.sh 2014-06-09 19:52:22 UTC (rev 7907) @@ -1,6 +0,0 @@ -#!/bin/bash -if [ "$HOSTNAME" == "midway001" ] -then - export GLOBUS_HOSTNAME=swift.rcc.uchicago.edu - export GLOBUS_TCP_PORT_RANGE=50000,51000 -fi; Deleted: branches/release-0.95/tests/sites/uc3/sanity.swift =================================================================== --- branches/release-0.95/tests/sites/uc3/sanity.swift 2014-06-09 17:03:04 UTC (rev 7906) +++ branches/release-0.95/tests/sites/uc3/sanity.swift 2014-06-09 19:52:22 UTC (rev 7907) @@ -1,11 +0,0 @@ -type file; - -app (file out, file err) remote_driver () -{ - date stdout=@filename(out) stderr=@filename(err); -} - -file driver_out ; -file driver_err ; - -(driver_out, driver_err) = remote_driver(); \ No newline at end of file Deleted: branches/release-0.95/tests/sites/uc3/sanity.timeout =================================================================== --- branches/release-0.95/tests/sites/uc3/sanity.timeout 2014-06-09 17:03:04 UTC (rev 7906) +++ branches/release-0.95/tests/sites/uc3/sanity.timeout 2014-06-09 19:52:22 UTC (rev 7907) @@ -1 +0,0 @@ -300 Modified: branches/release-0.95/tests/sites/uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/uc3/sites.template.xml 2014-06-09 17:03:04 UTC (rev 7906) +++ branches/release-0.95/tests/sites/uc3/sites.template.xml 2014-06-09 19:52:22 UTC (rev 7907) @@ -2,18 +2,20 @@ - + 10.00 10000 1 - 3600 - 00:30:00 + 1200 + 00:10:00 100 100 1 1 1 "Swift" + /tmp/swiftwork + nonshared /tmp/swiftwork From yadunandb at ci.uchicago.edu Mon Jun 9 16:55:46 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 9 Jun 2014 16:55:46 -0500 (CDT) Subject: [Swift-commit] r7908 - www/docs Message-ID: <20140609215546.4605D9D5A1@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-09 16:55:46 -0500 (Mon, 09 Jun 2014) New Revision: 7908 Modified: www/docs/index.php Log: Committing changes to docs page for localhost tutorial Modified: www/docs/index.php =================================================================== --- www/docs/index.php 2014-06-09 19:52:22 UTC (rev 7907) +++ www/docs/index.php 2014-06-09 21:55:46 UTC (rev 7908) @@ -48,9 +48,15 @@

     

    Swift Tutorials

    - The Swift tutorials give a quick hands-on taste of the key features of the language: + The Swift tutorials give a quick hands-on taste of the key features of the language. The following tutorial lets you try swift +on your local machine.

    + The following tutorials lets you try swift on different computation resources that you may have access to: +

    +
    • Swift on Clouds and Ad Hoc collections of workstations
    • Swift on OSG Connect
    • Swift on Crays
      • From yadunandb at ci.uchicago.edu Mon Jun 9 17:35:12 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 9 Jun 2014 17:35:12 -0500 (CDT) Subject: [Swift-commit] r7909 - SwiftTutorials Message-ID: <20140609223512.E29149D053@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-09 17:35:12 -0500 (Mon, 09 Jun 2014) New Revision: 7909 Added: SwiftTutorials/swift-localhost-tutorial/ Log: Adding directory for localhost tutorial From swift at ci.uchicago.edu Tue Jun 10 00:55:09 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 10 Jun 2014 00:55:09 -0500 (CDT) Subject: [Swift-commit] cog r3911 Message-ID: <20140610055509.47E0F8D00129@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3911 | hategan | 2014-06-10 00:53:56 -0500 (Tue, 10 Jun 2014) | 1 line added a name property to task handlers and make the AbstractionFactory set this property to the value of the requested provider ------------------------------------------------------------------------ Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/file/TaskHandlerImpl.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/file/TaskHandlerImpl.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/file/TaskHandlerImpl.java (working copy) @@ -526,4 +526,8 @@ protected FileResource getResource() { return this.fileResource; } + + public String getName() { + return "FileOperationTaskHandler"; + } } Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractionFactory.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractionFactory.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractionFactory.java (working copy) @@ -74,7 +74,9 @@ throws InvalidProviderException, ProviderMethodException { AbstractionProperties providerProps = AbstractionProperties .getProperties(provider); - return (TaskHandler) newObject(provider, type); + TaskHandler th = (TaskHandler) newObject(provider, type); + th.setName(provider); + return th; } /** Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/TaskHandlerSkeleton.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/TaskHandlerSkeleton.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/TaskHandlerSkeleton.java (working copy) @@ -17,7 +17,7 @@ public abstract class TaskHandlerSkeleton implements TaskHandler { private int type; - + public void cancel(Task task) throws InvalidSecurityContextException, TaskSubmissionException { cancel(task, null); @@ -30,4 +30,12 @@ public int getType() { return type; } + + public String toString() { + return "TaskHandler(type = " + getType() + ", provider = " + getName() + ")"; + } + + public void setName(String name) { + // ignored + } } Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileTransferTaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileTransferTaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileTransferTaskHandler.java (working copy) @@ -17,7 +17,7 @@ setType(TaskHandler.FILE_TRANSFER); } - protected String getName() { + public String getName() { return "FileTransferTaskHandler"; } Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileOperationTaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileOperationTaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/FileOperationTaskHandler.java (working copy) @@ -9,7 +9,6 @@ import java.util.ArrayList; import java.util.Collection; import java.util.HashMap; -import java.util.Iterator; import java.util.Map; import org.globus.cog.abstraction.impl.common.AbstractionFactory; @@ -21,10 +20,10 @@ import org.globus.cog.abstraction.interfaces.TaskHandler; public class FileOperationTaskHandler extends MultiplexingTaskHandler { - private Map mapping; + private Map mapping; public FileOperationTaskHandler() { - this.mapping = new HashMap(); + this.mapping = new HashMap(); setType(TaskHandler.FILE_OPERATION); } @@ -36,7 +35,7 @@ String provider = task.getService(0).getProvider().toLowerCase(); TaskHandler th = null; synchronized (this.mapping) { - th = (TaskHandler) this.mapping.get(provider); + th = this.mapping.get(provider); } if (th == null) { throw new TaskSubmissionException("Provider " + provider @@ -55,7 +54,7 @@ } String provider = task.getService(0).getProvider().toLowerCase(); synchronized (this.mapping) { - TaskHandler th = (TaskHandler) this.mapping.get(provider); + TaskHandler th = this.mapping.get(provider); if (th == null) { th = createTaskHandler(task); } @@ -99,18 +98,16 @@ public void remove(Task task) throws ActiveTaskException { String provider = task.getService(0).getProvider().toLowerCase(); - TaskHandler taskHandler = (TaskHandler) this.mapping.get(provider); + TaskHandler taskHandler = this.mapping.get(provider); if (taskHandler != null) { taskHandler.remove(task); } } - protected Collection getTasks(final TaskCollector collector) { + protected Collection getTasks(final TaskCollector collector) { // extract tasks from various TaskHandlers - ArrayList list = new ArrayList(); - Iterator i = this.mapping.values().iterator(); - while (i.hasNext()) { - TaskHandler handler = (TaskHandler) i.next(); + ArrayList list = new ArrayList(); + for (TaskHandler handler : mapping.values()) { list.addAll(collector.collect(handler)); } return list; @@ -132,4 +129,9 @@ this.mapping.put(provider.toLowerCase(), taskHandler); return taskHandler; } + + @Override + public String getName() { + return "FileOperationTaskHandler"; + } } \ No newline at end of file Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/ExecutionTaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/ExecutionTaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/ExecutionTaskHandler.java (working copy) @@ -185,7 +185,7 @@ return getTasksWithStatus(Status.CANCELED); } - public String toString() { + public String getName() { return "ExecutionTaskHandler"; } } Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/GenericTaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/GenericTaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/task/GenericTaskHandler.java (working copy) @@ -122,4 +122,8 @@ list.addAll(collector.collect(fileHandler)); return list; } + + public String getName() { + return "GenericTaskHandler"; + } } \ No newline at end of file Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/execution/TaskHandlerImpl.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/execution/TaskHandlerImpl.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/execution/TaskHandlerImpl.java (working copy) @@ -26,4 +26,8 @@ } protected abstract DelegatedTaskHandler newDelegatedTaskHandler(); + + public String toString() { + return "ExecutionTaskHandler(provider = " + getName() + ")"; + } } Index: modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractTaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractTaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/impl/common/AbstractTaskHandler.java (working copy) @@ -47,7 +47,11 @@ protected abstract DelegatedTaskHandler newDelegatedTaskHandler(int type) throws TaskSubmissionException; - protected abstract String getName(); + public abstract String getName(); + + public void setName(String name) { + // ignored for most handlers + } public void suspend(Task task) throws InvalidSecurityContextException, TaskSubmissionException { DelegatedTaskHandler dth = this.handleMap.get(task); Index: modules/abstraction-common/src/org/globus/cog/abstraction/interfaces/TaskHandler.java =================================================================== --- modules/abstraction-common/src/org/globus/cog/abstraction/interfaces/TaskHandler.java (revision 3910) +++ modules/abstraction-common/src/org/globus/cog/abstraction/interfaces/TaskHandler.java (working copy) @@ -65,6 +65,10 @@ * Returns the type of tasks handled by this TaskHandler */ public int getType(); + + public String getName(); + + public void setName(String provider); /** * Submits the given {@link Task}for execuiton. The Task is From swift at ci.uchicago.edu Tue Jun 10 01:00:16 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 10 Jun 2014 01:00:16 -0500 (CDT) Subject: [Swift-commit] cog r3917 Message-ID: <20140610060016.3D7898D00129@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3917 | hategan | 2014-06-10 00:58:48 -0500 (Tue, 10 Jun 2014) | 1 line removed WS provider ------------------------------------------------------------------------ Index: modules/provider-ws/project.properties =================================================================== --- modules/provider-ws/project.properties (revision 3916) +++ modules/provider-ws/project.properties (working copy) @@ -1,7 +0,0 @@ -module.name = provider-ws -long.name = Generic Web services provider for abstractions -version = 0.1 -project = Java CoG Kit -lib.deps = wsif.jar, wsdl4j.jar -debug = true - Index: modules/provider-ws/launchers.xml =================================================================== --- modules/provider-ws/launchers.xml (revision 3916) +++ modules/provider-ws/launchers.xml (working copy) @@ -1,6 +0,0 @@ - - - - - - Index: modules/provider-ws/dependencies.xml =================================================================== --- modules/provider-ws/dependencies.xml (revision 3916) +++ modules/provider-ws/dependencies.xml (working copy) @@ -1,8 +0,0 @@ - - - - - - - - Index: modules/provider-ws/lib/wsdl4j.jar =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Index: modules/provider-ws/lib/wsif.jar =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Index: modules/provider-ws/meta/description.txt =================================================================== --- modules/provider-ws/meta/description.txt (revision 3916) +++ modules/provider-ws/meta/description.txt (working copy) @@ -1,4 +0,0 @@ -The provider-ws module is a component of the abstractions framework. -It provides the implementation for the execution task handler -supporting invocations of generic Web services using the Web services -invication framework (WSIF). Index: modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationThread.java =================================================================== --- modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationThread.java (revision 3916) +++ modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationThread.java (working copy) @@ -1,270 +0,0 @@ -// ---------------------------------------------------------------------- -//This code is developed as part of the Java CoG Kit project -//The terms of the license can be found at http://www.cogkit.org/license -//This message may not be removed or altered. -//---------------------------------------------------------------------- - -package org.globus.cog.abstraction.impl.invocation.ws; - -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.Map; - -import javax.wsdl.Definition; -import javax.wsdl.Input; -import javax.wsdl.Operation; -import javax.wsdl.Output; -import javax.wsdl.Part; -import javax.wsdl.PortType; -import javax.wsdl.Service; -import javax.xml.namespace.QName; - -import org.apache.log4j.Logger; -import org.apache.wsif.WSIFException; -import org.apache.wsif.WSIFMessage; -import org.apache.wsif.WSIFOperation; -import org.apache.wsif.WSIFPort; -import org.apache.wsif.WSIFService; -import org.apache.wsif.WSIFServiceFactory; -import org.apache.wsif.util.WSIFUtils; -import org.globus.cog.abstraction.impl.common.StatusImpl; -import org.globus.cog.abstraction.interfaces.Status; -import org.globus.cog.abstraction.interfaces.Task; -import org.globus.cog.abstraction.interfaces.WSInvocationSpecification; - -public class InvocationThread extends Thread { - static Logger logger = Logger.getLogger(InvocationThread.class.getName()); - private Task task = null; - - public InvocationThread(Task task) { - this.task = task; - } - - public void run() { - try { - this.task.setStatus(Status.ACTIVE); - org.globus.cog.abstraction.interfaces.Service service = this.task - .getService(0); - WSInvocationSpecification spec = (WSInvocationSpecification) this.task - .getSpecification(); - String method = spec.getMethod(); - String args[] = spec.getArgumentsAsArray(); - HashMap map = invokeMethod(service.getServiceContact().getContact() - + "?wsdl", method, null, null, null, "", args, 0); - String output = null; - for (Iterator iterator = map.keySet().iterator(); iterator - .hasNext();) { - String name = (String) iterator.next(); - if (output == null) { - output = map.get(name) + "\n"; - } else { - output += map.get(name) + "\n"; - } - } - this.task.setStdOutput(output); - this.task.setStatus(Status.COMPLETED); - } catch (Exception e) { - Status newStatus = new StatusImpl(); - Status oldStatus = this.task.getStatus(); - newStatus.setPrevStatusCode(oldStatus.getStatusCode()); - newStatus.setStatusCode(Status.FAILED); - newStatus.setException(e); - this.task.setStatus(newStatus); - } - } - - private HashMap invokeMethod(String wsdlLocation, String operationName, - String inputName, String outputName, String portName, - String protocol, String[] args, int argShift) throws Exception { - String serviceNS = null; - String serviceName = null; - String portTypeNS = null; - String portTypeName = null; - - logger.debug("Reading WSDL document from '" + wsdlLocation + "'"); - Definition def = WSIFUtils.readWSDL(null, wsdlLocation); - - Service service = WSIFUtils.selectService(def, serviceNS, serviceName); - - Map portTypes = WSIFUtils.getAllItems(def, "PortType"); - if (portTypes.size() > 1 && portName != null) { - for (Iterator i = portTypes.keySet().iterator(); i.hasNext();) { - QName qn = (QName) i.next(); - if (portName.equals(qn.getLocalPart())) { - portTypeName = qn.getLocalPart(); - portTypeNS = qn.getNamespaceURI(); - break; - } - } - } - PortType portType = WSIFUtils.selectPortType(def, portTypeNS, - portTypeName); - - WSIFServiceFactory factory = WSIFServiceFactory.newInstance(); - WSIFService dpf = factory.getService(def, service, portType); - WSIFPort port = null; - if (portName == null) - port = dpf.getPort(); - else - port = dpf.getPort(portName); - - if (inputName == null && outputName == null) { - // retrieve list of operations - List operationList = portType.getOperations(); - - // try to find input and output names for the operation specified - boolean found = false; - for (Iterator i = operationList.iterator(); i.hasNext();) { - Operation op = (Operation) i.next(); - String name = op.getName(); - if (!name.equals(operationName)) { - continue; - } - if (found) { - throw new RuntimeException( - "Operation '" - + operationName - + "' is overloaded. " - + "Please specify the operation in the form " - + "'operationName:inputMessageName:outputMesssageName' to distinguish it"); - } - found = true; - Input opInput = op.getInput(); - inputName = (opInput.getName() == null) ? null : opInput - .getName(); - Output opOutput = op.getOutput(); - outputName = (opOutput.getName() == null) ? null : opOutput - .getName(); - } - } - - WSIFOperation operation = port.createOperation(operationName, - inputName, outputName); - WSIFMessage input = operation.createInputMessage(); - WSIFMessage output = operation.createOutputMessage(); - WSIFMessage fault = operation.createFaultMessage(); - - // retrieve list of names and types for input and names for output - List operationList = portType.getOperations(); - - // find portType operation to prepare in/out message w/ parts - boolean found = false; - String[] outNames = new String[0]; - Class[] outTypes = new Class[0]; - for (Iterator i = operationList.iterator(); i.hasNext();) { - Operation op = (Operation) i.next(); - String name = op.getName(); - if (!name.equals(operationName)) { - continue; - } - if (found) { - throw new RuntimeException( - "overloaded operations are not supported"); - } - found = true; - Input opInput = op.getInput(); - String[] inNames = new String[0]; - Class[] inTypes = new Class[0]; - if (opInput != null) { - List parts = opInput.getMessage().getOrderedParts(null); - unWrapIfWrappedDocLit(parts, name, def); - int count = parts.size(); - inNames = new String[count]; - inTypes = new Class[count]; - retrieveSignature(parts, inNames, inTypes); - } - for (int pos = 0; pos < inNames.length; ++pos) { - String arg = args[pos + argShift]; - Object value = null; - Class c = inTypes[pos]; - if (c.equals(String.class)) { - value = arg; - } else if (c.equals(Double.TYPE)) { - value = new Double(arg); - } else if (c.equals(Float.TYPE)) { - value = new Float(arg); - } else if (c.equals(Integer.TYPE)) { - value = new Integer(arg); - } else if (c.equals(Boolean.TYPE)) { - value = new Boolean(arg); - } else { - throw new RuntimeException("Cannot convert '" + arg - + "' to " + c); - } - - input.setObjectPart(inNames[pos], value); - } - - Output opOutput = op.getOutput(); - if (opOutput != null) { - List parts = opOutput.getMessage().getOrderedParts(null); - unWrapIfWrappedDocLit(parts, name + "Response", def); - int count = parts.size(); - outNames = new String[count]; - outTypes = new Class[count]; - retrieveSignature(parts, outNames, outTypes); - } - - } - if (!found) { - throw new RuntimeException("no operation " + operationName - + " was found in port type " + portType.getQName()); - } - - logger.debug("Executing operation " + operationName); - operation.executeRequestResponseOperation(input, output, fault); - - HashMap map = new HashMap(); - for (int pos = 0; pos < outNames.length; ++pos) { - String name = outNames[pos]; - map.put(name, output.getObjectPart(name)); - } - - return map; - } - - private void retrieveSignature(List parts, String[] names, Class[] types) { - // get parts in correct order - for (int i = 0; i < names.length; ++i) { - Part part = (Part) parts.get(i); - names[i] = part.getName(); - QName partType = part.getTypeName(); - if (partType == null) { - partType = part.getElementName(); - } - if (partType == null) { - throw new RuntimeException("part " + names[i] - + " must have type name declared"); - } - // only limited number of types is supported - // cheerfully ignoring schema namespace ... - String s = partType.getLocalPart(); - if ("string".equals(s)) { - types[i] = String.class; - } else if ("double".equals(s)) { - types[i] = Integer.TYPE; - } else if ("float".equals(s)) { - types[i] = Float.TYPE; - } else if ("int".equals(s)) { - types[i] = Integer.TYPE; - } else if ("boolean".equals(s)) { - types[i] = Boolean.TYPE; - } else { - throw new RuntimeException("part type " + partType - + " not supported in this sample"); - } - } - } - - private void unWrapIfWrappedDocLit(List parts, String operationName, - Definition def) throws WSIFException { - Part p = WSIFUtils.getWrappedDocLiteralPart(parts, operationName); - if (p != null) { - List unWrappedParts = WSIFUtils.unWrapPart(p, def); - parts.remove(p); - parts.addAll(unWrappedParts); - } - } - -} \ No newline at end of file Index: modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/TaskHandlerImpl.java =================================================================== --- modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/TaskHandlerImpl.java (revision 3916) +++ modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/TaskHandlerImpl.java (working copy) @@ -1,247 +0,0 @@ -// ---------------------------------------------------------------------- -// This code is developed as part of the Java CoG Kit project -// The terms of the license can be found at http://www.cogkit.org/license -// This message may not be removed or altered. -// ---------------------------------------------------------------------- - -package org.globus.cog.abstraction.impl.invocation.ws; - -import java.util.ArrayList; -import java.util.Collection; -import java.util.Hashtable; -import java.util.Vector; - -import org.globus.cog.abstraction.impl.common.StatusEvent; -import org.globus.cog.abstraction.impl.common.task.ActiveTaskException; -import org.globus.cog.abstraction.impl.common.task.IllegalSpecException; -import org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException; -import org.globus.cog.abstraction.impl.common.task.InvalidServiceContactException; -import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; -import org.globus.cog.abstraction.interfaces.DelegatedTaskHandler; -import org.globus.cog.abstraction.interfaces.Status; -import org.globus.cog.abstraction.interfaces.StatusListener; -import org.globus.cog.abstraction.interfaces.Task; -import org.globus.cog.abstraction.interfaces.TaskHandler; - -/** - * Provides generic TaskHandler s for job submission task. - */ -public class TaskHandlerImpl implements TaskHandler, StatusListener { - private Vector submittedList = null; - private Vector activeList = null; - private Vector suspendedList = null; - private Vector resumedList = null; - private Vector failedList = null; - private Vector canceledList = null; - private Vector completedList = null; - private Hashtable handleMap = null; - private int type; - - public TaskHandlerImpl() { - this.submittedList = new Vector(); - this.activeList = new Vector(); - this.suspendedList = new Vector(); - this.resumedList = new Vector(); - this.failedList = new Vector(); - this.canceledList = new Vector(); - this.completedList = new Vector(); - this.handleMap = new Hashtable(); - this.type = TaskHandler.EXECUTION; - } - - public void setType(int type) { - this.type = type; - } - - public int getType() { - return this.type; - } - - public void submit(Task task) throws IllegalSpecException, - InvalidSecurityContextException, InvalidServiceContactException, - TaskSubmissionException { - if (task.getStatus().getStatusCode() != Status.UNSUBMITTED) { - throw new TaskSubmissionException( - "TaskHandler can only handle unsubmitted tasks"); - } - if (task.getType() != Task.WS_INVOCATION) { - throw new TaskSubmissionException( - getName() - + " execution task handler can only handle WS invocation tasks"); - } - int type = task.getType(); - DelegatedTaskHandler dth = newDelegatedTaskHandler(); - task.addStatusListener(this); - this.handleMap.put(task, dth); - dth.submit(task); - } - - protected DelegatedTaskHandler newDelegatedTaskHandler() { - return new InvocationTaskHandler(); - } - - protected String getName() { - return "Invocation"; - } - - public void suspend(Task task) throws InvalidSecurityContextException, - TaskSubmissionException { - DelegatedTaskHandler dth = (DelegatedTaskHandler) this.handleMap - .get(task); - if (dth != null) { - dth.suspend(); - } - } - - public void resume(Task task) throws InvalidSecurityContextException, - TaskSubmissionException { - DelegatedTaskHandler dth = (DelegatedTaskHandler) this.handleMap - .get(task); - if (dth != null) { - dth.resume(); - } - } - - public void cancel(Task task) throws InvalidSecurityContextException, - TaskSubmissionException { - DelegatedTaskHandler dth = (DelegatedTaskHandler) this.handleMap - .get(task); - if (dth != null) { - dth.cancel(); - } else { - task.setStatus(Status.CANCELED); - } - } - - public void remove(Task task) throws ActiveTaskException { - if (!handleMap.containsKey(task)) { - return; - } - int status = task.getStatus().getStatusCode(); - if ((status == Status.ACTIVE)) { - throw new ActiveTaskException( - "Cannot remove an active or suspended Task"); - } else { - // might cause problems - // task.removeStatusListener(this); - this.failedList.remove(task); - this.canceledList.remove(task); - this.completedList.remove(task); - this.suspendedList.remove(task); - this.submittedList.remove(task); - this.handleMap.remove(task); - - /* - * this is required because statusChange listeners are invoked in a - * single thread So if one listener that wants to remove a task from - * the handler is invoked before the listener implemented by this - * object, the task will be be in the activeList and will not be - * actually removed. - * - * To solve this we remove tasks from active list too - */ - this.activeList.remove(task); - } - } - - /** return a collection of all tasks submitted to the handler */ - public Collection getAllTasks() { - try { - return new ArrayList(handleMap.keySet()); - } catch (Exception e) { - return null; - } - } - - /** return a collection of active tasks */ - public Collection getActiveTasks() { - return new ArrayList(this.activeList); - } - - /** return a collection of failed tasks */ - public Collection getFailedTasks() { - return new ArrayList(this.failedList); - } - - /** return a collection of completed tasks */ - public Collection getCompletedTasks() { - return new ArrayList(this.completedList); - } - - /** return a collection of suspended tasks */ - public Collection getSuspendedTasks() { - return new ArrayList(this.suspendedList); - } - - /** return a collection of resumed tasks */ - public Collection getResumedTasks() { - return new ArrayList(this.resumedList); - } - - /** return a collection of canceled tasks */ - public Collection getCanceledTasks() { - return new ArrayList(this.canceledList); - } - - public void statusChanged(StatusEvent event) { - Task task = (Task) event.getSource(); - Status status = event.getStatus(); - int prevStatus = status.getPrevStatusCode(); - int curStatus = status.getStatusCode(); - switch (prevStatus) { - case Status.SUBMITTED: - this.submittedList.remove(task); - break; - case Status.ACTIVE: - this.activeList.remove(task); - break; - case Status.SUSPENDED: - this.suspendedList.remove(task); - break; - case Status.RESUMED: - this.resumedList.remove(task); - break; - case Status.FAILED: - this.failedList.remove(task); - break; - case Status.CANCELED: - this.canceledList.remove(task); - break; - case Status.COMPLETED: - this.completedList.remove(task); - break; - default: - break; - } - if (task != null) { - switch (curStatus) { - case Status.SUBMITTED: - this.submittedList.add(task); - break; - case Status.ACTIVE: - this.activeList.add(task); - break; - case Status.SUSPENDED: - this.suspendedList.add(task); - break; - case Status.RESUMED: - this.resumedList.add(task); - break; - case Status.FAILED: - this.failedList.add(task); - this.handleMap.remove(task); - break; - case Status.CANCELED: - this.canceledList.add(task); - this.handleMap.remove(task); - break; - case Status.COMPLETED: - this.completedList.add(task); - this.handleMap.remove(task); - break; - default: - break; - } - } - } -} \ No newline at end of file Index: modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationTaskHandler.java =================================================================== --- modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationTaskHandler.java (revision 3916) +++ modules/provider-ws/src/org/globus/cog/abstraction/impl/invocation/ws/InvocationTaskHandler.java (working copy) @@ -1,67 +0,0 @@ -// ---------------------------------------------------------------------- -//This code is developed as part of the Java CoG Kit project -//The terms of the license can be found at http://www.cogkit.org/license -//This message may not be removed or altered. -//---------------------------------------------------------------------- - -package org.globus.cog.abstraction.impl.invocation.ws; - -import org.apache.log4j.Logger; -import org.globus.cog.abstraction.impl.common.StatusImpl; -import org.globus.cog.abstraction.impl.common.task.IllegalSpecException; -import org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException; -import org.globus.cog.abstraction.impl.common.task.InvalidServiceContactException; -import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; -import org.globus.cog.abstraction.interfaces.DelegatedTaskHandler; -import org.globus.cog.abstraction.interfaces.Status; -import org.globus.cog.abstraction.interfaces.Task; - -public class InvocationTaskHandler implements DelegatedTaskHandler { - private static Logger logger = Logger - .getLogger(InvocationTaskHandler.class); - - private Task task = null; - - public void submit(Task task) throws IllegalSpecException, - InvalidSecurityContextException, InvalidServiceContactException, - TaskSubmissionException { - if (this.task != null) { - throw new TaskSubmissionException( - "InvocationTaskHandler cannot handle two active tasks simultaneously"); - } else { - this.task = task; - InvocationThread thread = new InvocationThread(this.task); - try { - // check if the task has not been canceled after it was - // submitted for execution - if (this.task.getStatus().getStatusCode() == Status.UNSUBMITTED) { - this.task.setStatus(Status.SUBMITTED); - thread.start(); - } - } catch (Exception e) { - e.printStackTrace(); - Status newStatus = new StatusImpl(); - Status oldStatus = this.task.getStatus(); - newStatus.setPrevStatusCode(oldStatus.getStatusCode()); - newStatus.setStatusCode(Status.FAILED); - newStatus.setException(e); - this.task.setStatus(newStatus); - } - - } - } - - public void suspend() throws InvalidSecurityContextException, - TaskSubmissionException { - } - - public void resume() throws InvalidSecurityContextException, - TaskSubmissionException { - } - - public void cancel() throws InvalidSecurityContextException, - TaskSubmissionException { - - this.task.setStatus(Status.CANCELED); - } -} \ No newline at end of file Index: modules/provider-ws/resources/cog-provider.properties =================================================================== --- modules/provider-ws/resources/cog-provider.properties (revision 3916) +++ modules/provider-ws/resources/cog-provider.properties (working copy) @@ -1,5 +0,0 @@ -provider=ws -sandbox=false -executionTaskHandler=org.globus.cog.abstraction.impl.invocation.ws.TaskHandlerImpl -securityContext=org.globus.cog.abstraction.impl.common.task.SecurityContextImpl - Index: modules/provider-ws/etc/MANIFEST.MF.tail =================================================================== --- modules/provider-ws/etc/MANIFEST.MF.tail (revision 3916) +++ modules/provider-ws/etc/MANIFEST.MF.tail (working copy) @@ -1 +0,0 @@ - Index: modules/provider-ws/etc/MANIFEST.MF.head =================================================================== --- modules/provider-ws/etc/MANIFEST.MF.head (revision 3916) +++ modules/provider-ws/etc/MANIFEST.MF.head (working copy) @@ -1 +0,0 @@ -Manifest-Version: 1.0 Index: modules/provider-ws/build.xml =================================================================== --- modules/provider-ws/build.xml (revision 3916) +++ modules/provider-ws/build.xml (working copy) @@ -1,152 +0,0 @@ - - - - - - - - - - - - - - - - Available targets: - help: - prints out this help message - - dist: - creates a distribution directory of the - ${project} ${long.name} - - jar: - creates a jar file for the ${project} ${long.name} - named ${jar.filename} - - javadoc: - creates the documentation - - clean: - removes the compiled classes - - distclean: - deletes the distribution directory - - all: - dist and javadoc - - deploy.webstart: - deploys the module as a webstart application - - dist.joint: - builds everything into one jar file. Should only - be used globally (from all) - - fixeol: - change newlines to the unix standard - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - From swift at ci.uchicago.edu Tue Jun 10 01:05:09 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 10 Jun 2014 01:05:09 -0500 (CDT) Subject: [Swift-commit] cog r3918 Message-ID: <20140610060510.322A98D00129@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3918 | hategan | 2014-06-10 01:02:50 -0500 (Tue, 10 Jun 2014) | 1 line shell provider ------------------------------------------------------------------------ Index: modules/provider-localscheduler/shell-provider.README =================================================================== --- modules/provider-localscheduler/shell-provider.README (revision 0) +++ modules/provider-localscheduler/shell-provider.README (revision 3918) @@ -0,0 +1,276 @@ +=================== += I. Introduction = +=================== + +This file contains information about the shell provider. The shell +provider is a CoG provider that allows interfacing with cutsom queuing +systems for which a provider is not otherwise available. + +Names contained within angle brackets are meant to be replaced with +specific strings or values. For example, if you want to name your +provider "PBS" then replace all occurrences of with +"PBS". + +There are five basic ingredients to a shell provider: + + 1. A file that informs the job abstraction code about the existence + of the new provider (cog-provider.properties) + + 2. A file that contains settings for the new provider + (provider-.properties) + + 3. A submit script that receives job information and is responsible + for starting the job. + + 4. A cancel script that can be used to cancel a submitted job + + 5. A status script that provides information about the status of + jobs + +For a sample implementation, see the 'examples' directory. + +============= += II. Setup = +============= + +1. cog-provider.properties +-------------------------- + +The first step is creating the cog-provider.properties file. For each +provider that you want to add, make sure that the file contains the +following lines: + + provider= + sandbox=false + executionTaskHandler=org.globus.cog.abstraction.impl.scheduler.shell.execution.TaskHandlerImpl + securityContext=org.globus.cog.abstraction.impl.common.task.SecurityContextImpl + +You can add multiple sets of these, one for each provider you want to +implement. The cog-provider.properties file must be put in a directory +(or .jar file) that is included in Java's class path at run-time. If +using any of the CoG modules in a normal deployment or Swift, the +dist//etc directory is automatically included in the class +path. + +If you wanted this provider to be available on a remote site accessed by +coasters, you could put this file in a directory that would be added to +the CLASSPATH environment variable in the shell profile (although at the +time of this writing the author is not entirely sure that CLASSPATH is +used by the coaster bootstrap process). + + +2. provider-.properties +------------------------------------- + +This file must contain the following entries: + + submit.command= + cancel.command= + poll.command= + +Optionally, it could contain the following property: + + poll.interval= + +The poll interval indicates the interval at which the status of jobs is +checked. The check is done regularly at this interval for all jobs +sumbitted through this provider. The default is 5 seconds. + + +3. The submit script +-------------------- + +This script is responsible for submitting the actual job. It receives +input on STDIN. It must produce its normal output on STDOUT and if an +error occurs, it should go on STDERR. + +Following are the syntactic rules for the input: + - empty lines must be ignored + - lines starting with the '#' character must also be ignored + - each other line is a "key=value" pair + - values can contain unescaped '=' symbols, so when parsing lines, + the first equal sign signifies the end of the key + - new lines are escaped using '\n' and backslashes are escaped + as '\\'; there are no other escape codes + - for certain keys, multiple lines with the same key can appear + in the input; this is the way to encode a list + - keys appear in the input in a well defined order; however, some + keys can be missing, indicating that the corresponding value is + not set + +The following keys (in the specified order) can appear in the input. +Optional keys are marked with '?' (although no '?' will appear in the +actual input to the submit script). Similarly, lists (i.e. cases when +the key can appear zero or more times) are marked with '*'. Literals +are single-quoted. The pipe ('|') represents mutually exclusive choices. + +?directory= + if specified, run the job in + +executable= + this is the path to the executable + +*arg= + one of the arguments to the executable + +?stdin.location='local'|'remote' + indicates that the executable should be fed a file to its STDIN and + whether this file is on the submit side ('local') or on the node + where the job will run ('remote'). If this key is present, then + the following key is mandatory. + + Practically, if the file is located on a shared file system + accessible (and mounted in the same place) by both the submit + host and the compute node, then treating 'remote' as 'local' + and the other way around should produce no visible difference. + + For the purpose of implementing a provider that can be used to + control coaster blocks, assuming a shared file system and + implementing this in the easiest way is a reasonable way to go. + +?stdin.path= + The path to the file to be fed to the STDIN of the job. + +?stdout.location='local'|'remote'|'tmp' + specifies where the process STDOUT should be redirected. With + the exception of 'tmp', the same considerations as in the case + of stdin.location above apply. + + 'tmp' is a special case. It says that the submit script must + re-direct the job stdout to some temporary file AND return + (on STDOUT) the path to this file in the following form: + + stdout.path= + + if 'tmp' is specified, the following key (stdout.path) will + not be present + +?stdout.path= + where to re-direct the job stdout if stdout.location is 'local' + or 'remote' + +?stderr.location='local'|'remote'|'tmp' + see stdout.location + +?stderr.path= + see stdout.path + +*env.= + Specifies an environment variable that muts be available in the + job environment + +*attr.= + These are miscellaneous job attributes. Their interpretation + is up to the provider. Some example of attributes include + 'maxwalltime', 'queue', 'project', etc. Unknown attributes + should be ignored. + + Coasters will set the following attributes: + maxwalltime= - obvious + count= - the number of process copies to start + +----- +Small aside: file staging/cleanup is optional. If not implemented, +an error should be produced. +----- + +*stagein.source= + Specifies a stage-in request. If present, it will be followed + by stagein.destination. + represents the local file that must be copied to the + compute node. + +*stagein.destination= + Specifies where (on the compute node) the most recent + stagein.source should be copied + +*stageout.source= + Specifies a stage-out request. If present, it will be followed + by stageout.destination and stageout.mode. + represents the remote file that must be copied from the + compute node to the submit node. + +*stageout.destination= + Specifies where (on the submit node) the most recent + stageout.source should be copied + +*stagout.mode= + Specifies under what circumstances the most recent stagout + request should be fulfilled. The is a bit-wise OR + of the following possible values: + + 1 - always: stage out the file whether the job succeeds or + not. If the file cannot be staged out, produce an error + 2 - if present: stage out the file only if it can be found + 3 - on error: only attempt to stage out the file if the job + fails; produce an error if the job failed and the file cannot + be staged-out + 4 - on success: like "on error" except if the job succeeds. + +*cleanup= + A list of files and or directories to be deleted after the + job is done and the stage out process is completed. Reasonable + implementations could ensure that the cleanup paths are + limited to the job directory + +The submit script, after starting or queuing the job, must reply, +on STDOUT, with "jobid=", where is a handle that +can be used to subsequently identify the job for purposes of +cancelling or querying its status. If the submission is successful, +the implementation should return an exit code of 0. + +If the implementation finds that the specification is erroneous, or +if any errors occur submitting the job, an error message should be +printed on STDERR and an exit code != 0 should be produced. + +Here is an example: + STDIN: + directory=/tmp + executable=/bin/ls + arg=-al + arg=*.txt + stdout.location=tmp + + A local execution submit script could perform the following: + cd /tmp + /bin/ls -al *.txt 1>/tmp/tempx0001 & + PID=$! + echo "stdout.path=/tmp/tempx0001" + echo "jobid=$PID" + + +4. The cancel script +-------------------- + +The cancel script must cancel the job with the passed to +it as the firs command-line argument. + + +5. The status script +-------------------- + +The status script implements a mechanism for querying the status +of jobs. It receives a list of s on the command line, one +in each argument, and returns, if successful, on STDOUT, a list +of lines, one for each job, of the following format: + + 'Q'|'R'|'C'|'F' [ []] + +The meaning of the second column is as follows: + 'Q': the job is queued + 'R': the job is running + 'C': the job is completed + optionally, an could be present + 'F': the job is failed + optionally, an should be present, along possibly + with an + +There is some level of ambiguity about the interpretation of non-zero +exit codes. An implementation can chose to consider all jobs that +do not run any more as completed, and use the exit code as an +indication of whether the completion was successful or not. Such +an implementation might reserve the failed state for cases in +which the job has failed to run due to queuing system conditions, +and provider, in an appropriate explanation of the +failure condition. Other implementations may chose to mark all jobs +completed with non-zero exit codes as failed. Index: modules/provider-localscheduler/examples/shell-provider/ststat =================================================================== --- modules/provider-localscheduler/examples/shell-provider/ststat (revision 0) +++ modules/provider-localscheduler/examples/shell-provider/ststat (revision 3918) @@ -0,0 +1,12 @@ +#!/bin/bash + +while [ "$1" != "" ]; do + if [ $(pidof $1) ]; then + echo $1 R + else + # Assume succes if the process is not running + # This is just a simple example + echo $1 C 0 + fi + shift 1 +done \ No newline at end of file Property changes on: modules/provider-localscheduler/examples/shell-provider/ststat ___________________________________________________________________ Added: svn:executable + * Index: modules/provider-localscheduler/examples/shell-provider/provider-shelltest.properties =================================================================== --- modules/provider-localscheduler/examples/shell-provider/provider-shelltest.properties (revision 0) +++ modules/provider-localscheduler/examples/shell-provider/provider-shelltest.properties (revision 3918) @@ -0,0 +1,3 @@ +submit.command=/path/to/stsubmit +cancel.command=/path/to/stcancel +poll.command=/path/to/ststat \ No newline at end of file Index: modules/provider-localscheduler/examples/shell-provider/stsubmit =================================================================== --- modules/provider-localscheduler/examples/shell-provider/stsubmit (revision 0) +++ modules/provider-localscheduler/examples/shell-provider/stsubmit (revision 3918) @@ -0,0 +1,92 @@ +#!/bin/bash + + +rm -f /tmp/stsubmit +EXECUTABLE= +DIR= +ARGS= +STDOUTLOC= +STDOUTPATH= +STDERRLOC= +STDERRPATH= +STDINLOC= +STDINPATH= +STDIN= +STDOUT= +STDERR= + +while read LINE; do + echo $LINE >>/tmp/stsubmit + case $LINE in + executable=*) + EXECUTABLE=${LINE#executable=} + ;; + directory=*) + DIR=${LINE#directory=} + ;; + arg=*) + ARGS="$ARGS ${LINE#arg=}" + ;; + attr.*) + # ignore attributes + ;; + stdin.location=*) + STDINLOC=${LINE#stdin.location=} + ;; + stdin.path=*) + STDINPATH=${LINE#stdin.path=} + ;; + stdout.location=*) + STDOUTLOC=${LINE#stdout.location=} + ;; + stdout.path=*) + STDOUTPATH=${LINE#stdout.path=} + ;; + stderr.location=*) + STDERRLOC=${LINE#stderr.location=} + ;; + stderr.path=*) + STDERRPATH=${LINE#stderr.path=} + ;; + env.*) + LINE2=${LINE#env.} + # split on '=' + ELS=(${LINE2//=/}) + NAME=${ELS[0]} + VALUE=${ELS[1]} + export $NAME=$VALUE + ;; + *) + echo "Don't know how to interpret line: $LINE" >&2 + exit 2 + esac +done < /dev/stdin + +if [ "$STDOUTLOC" == "tmp" ]; then + STDOUTPATH=$(mktemp) + echo "stdout.path=$STDOUTPATH" +fi +if [ "$STDOUTPATH" != "" ]; then + STDOUT="1> $STDOUTPATH" +fi + +if [ "$STDERRLOC" == "tmp" ]; then + STDERRPATH=$(mktemp) + echo "stderr.path=$STDERRPATH" +fi +if [ "$STDERRPATH" != "" ]; then + STDERR="2> $STDERRPATH" +fi + +if [ "$STDINLOC" != "" ]; then + STDIN="< $STDINLOC" +fi + +CMD="$EXECUTABLE $ARGS $STDIN $STDOUT $STDERR" + +cd $DIR +$CMD & + +echo "jobid=$!" + +exit 0 \ No newline at end of file Property changes on: modules/provider-localscheduler/examples/shell-provider/stsubmit ___________________________________________________________________ Added: svn:executable + * Index: modules/provider-localscheduler/examples/shell-provider/cog-provider.properties =================================================================== --- modules/provider-localscheduler/examples/shell-provider/cog-provider.properties (revision 0) +++ modules/provider-localscheduler/examples/shell-provider/cog-provider.properties (revision 3918) @@ -0,0 +1,4 @@ +provider=shelltest +sandbox=false +executionTaskHandler=org.globus.cog.abstraction.impl.scheduler.shell.execution.TaskHandlerImpl +securityContext=org.globus.cog.abstraction.impl.common.task.SecurityContextImpl Index: modules/provider-localscheduler/examples/shell-provider/stcancel =================================================================== --- modules/provider-localscheduler/examples/shell-provider/stcancel (revision 0) +++ modules/provider-localscheduler/examples/shell-provider/stcancel (revision 3918) @@ -0,0 +1,3 @@ +#!/bin/bash + +kill $1 Property changes on: modules/provider-localscheduler/examples/shell-provider/stcancel ___________________________________________________________________ Added: svn:executable + * Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/QueuePoller.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/QueuePoller.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/QueuePoller.java (revision 3918) @@ -0,0 +1,107 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Oct 11, 2005 + */ +package org.globus.cog.abstraction.impl.scheduler.shell; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; + +import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller; +import org.globus.cog.abstraction.impl.scheduler.common.Job; + +public class QueuePoller extends AbstractQueuePoller { + public static final Logger logger = Logger.getLogger(QueuePoller.class); + + + public QueuePoller(Properties properties) { + super(properties.getName() + "provider queue poller", properties); + } + + private static String[] CMDARRAY; + + protected synchronized String[] getCMDArray() { + String[] cmda = new String[1 + getJobs().size()]; + cmda[0] = getProperties().getPollCommand(); + int i = 1; + for (Job j : getJobs().values()) { + cmda[i++] = j.getJobID(); + } + return cmda; + } + + protected void processStdout(InputStream is) throws IOException { + BufferedReader br = new BufferedReader(new InputStreamReader(is)); + String line; + do { + line = br.readLine(); + if (line != null) { + try { + String[] el = line.trim().split("\\s+", 4); + String jobid = el[0]; + String message = null; + + if (el.length > 3) { + message = el[3]; + } + + Job job = getJob(jobid); + if (job == null) { + logger.warn("Received status for an unknown job: '" + jobid + "'"); + } + + if (el[1].length() != 1) { + job.setMessage("Received unknown status code: '" + el[1] + "'"); + job.setState(Job.STATE_FAILED); + } + + char status = el[1].charAt(0); + if (logger.isDebugEnabled()) { + logger.debug("Status for " + jobid + " is " + status); + } + + if (el.length > 2) { + int exitCode = Integer.parseInt(el[2]); + job.setExitcode(exitCode); + } + if (status == 'Q') { + job.setState(Job.STATE_QUEUED); + } + else if (status == 'R') { + job.setState(Job.STATE_RUNNING); + } + else if (status == 'C') { + job.setState(Job.STATE_DONE); + } + else if (status == 'F') { + if (message != null) { + job.setMessage(message); + } + else { + job.setMessage("Job failed"); + } + job.setState(Job.STATE_FAILED); + } + + } + catch (Exception e) { + logger.warn("Exception caught while handling " + + getProperties().getPollCommandName() + + " output: " + line, e); + } + } + } while (line != null); + } + + protected void processStderr(InputStream is) throws IOException { + // not used + } +} Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/TaskHandlerImpl.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/TaskHandlerImpl.java (revision 3918) @@ -0,0 +1,50 @@ + +// ---------------------------------------------------------------------- +// This code is developed as part of the Java CoG Kit project +// The terms of the license can be found at http://www.cogkit.org/license +// This message may not be removed or altered. +// ---------------------------------------------------------------------- + +package org.globus.cog.abstraction.impl.scheduler.shell.execution; + +import org.globus.cog.abstraction.impl.scheduler.shell.Properties; +import org.globus.cog.abstraction.interfaces.DelegatedTaskHandler; + +public class TaskHandlerImpl extends + org.globus.cog.abstraction.impl.common.execution.TaskHandlerImpl { + private Properties props; + private String name; + + protected DelegatedTaskHandler newDelegatedTaskHandler() { + return new JobSubmissionTaskHandler(props); + } + + public String getName() { + return name; + } + + public void setName(String name) { + if (this.name != null) { + throw new IllegalStateException("Name is already set to '" + name + "'"); + } + this.name = name; + props = Properties.getProperties(name); + validateProperties(); + } + + private void validateProperties() { + if (props.getSubmitCommand() == null) { + throw new IllegalArgumentException("Invalid configuration for provider '" + name + "'. No submit command specified."); + } + + if (props.getRemoveCommand() == null) { + throw new IllegalArgumentException("Invalid configuration for provider '" + name + "'. No cancel command specified."); + } + + if (props.getPollCommand() == null) { + throw new IllegalArgumentException("Invalid configuration for provider '" + name + "'. No poll command specified."); + } + } + + +} \ No newline at end of file Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/JobSubmissionTaskHandler.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/JobSubmissionTaskHandler.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/execution/JobSubmissionTaskHandler.java (revision 3918) @@ -0,0 +1,26 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +package org.globus.cog.abstraction.impl.scheduler.shell.execution; + +import org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler; +import org.globus.cog.abstraction.impl.scheduler.shell.Properties; +import org.globus.cog.abstraction.impl.scheduler.shell.ShellExecutor; +import org.globus.cog.abstraction.interfaces.Task; + +public class JobSubmissionTaskHandler extends AbstractJobSubmissionTaskHandler { + private Properties props; + + public JobSubmissionTaskHandler(Properties props) { + this.props = props; + } + + protected AbstractExecutor newExecutor(Task task, + AbstractJobSubmissionTaskHandler th) { + return new ShellExecutor(props, task, th); + } +} \ No newline at end of file Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/ShellExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/ShellExecutor.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/ShellExecutor.java (revision 3918) @@ -0,0 +1,345 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Oct 11, 2005 + */ +package org.globus.cog.abstraction.impl.scheduler.shell; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.io.Reader; +import java.io.Writer; +import java.text.DecimalFormat; +import java.text.NumberFormat; +import java.util.EnumSet; + +import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractProperties; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller; +import org.globus.cog.abstraction.impl.scheduler.common.Job; +import org.globus.cog.abstraction.impl.scheduler.common.ProcessException; +import org.globus.cog.abstraction.impl.scheduler.common.ProcessListener; +import org.globus.cog.abstraction.interfaces.FileLocation; +import org.globus.cog.abstraction.interfaces.JobSpecification; +import org.globus.cog.abstraction.interfaces.StagingSet; +import org.globus.cog.abstraction.interfaces.StagingSetEntry; +import org.globus.cog.abstraction.interfaces.StagingSetEntry.Mode; +import org.globus.cog.abstraction.interfaces.Task; +import org.globus.gsi.gssapi.auth.AuthorizationException; + +public class ShellExecutor extends AbstractExecutor { + public static final Logger logger = Logger.getLogger(ShellExecutor.class); + + private Properties props; + private static volatile int idCounter; + private String name, jobid; + private boolean active; + + public ShellExecutor(Properties props, Task task, ProcessListener listener) { + super(task, listener); + this.props = props; + name = "cog-" + IDF.format(idCounter++); + } + + private static NumberFormat IDF = new DecimalFormat("000000"); + + protected void writeSpec(String name, String value, Writer wr) + throws IOException { + if (value != null) { + wr.write(name); + wr.write('='); + wr.write(String.valueOf(value)); + wr.write('\n'); + } + } + + private static class ProcessStreams { + private Process process; + public Writer stdin; + private Reader stdout, stderr; + private StringBuilder out, err; + + public ProcessStreams(Process process) { + this.process = process; + this.stdin = new OutputStreamWriter(process.getOutputStream()); + this.stdout = new InputStreamReader(process.getInputStream()); + this.stderr = new InputStreamReader(process.getErrorStream()); + out = new StringBuilder(); + err = new StringBuilder(); + } + } + + @Override + public void start() throws AuthorizationException, IOException, ProcessException { + Process p = Runtime.getRuntime().exec(props.getSubmitCommand()); + ProcessStreams ps = new ProcessStreams(p); + + JobSpecification spec = this.getSpec(); + + // Basic stuff + if (!spec.getExecutableLocation().equals(FileLocation.REMOTE)) { + throw new IllegalArgumentException("Only remote executables supported"); + } + writeSpec("directory", escape(spec.getDirectory()), ps.stdin); + writeSpec("executable", escape(spec.getExecutable()), ps.stdin); + if (spec.getArgumentsAsList() != null) { + for (String arg : spec.getArgumentsAsList()) { + writeSpec("arg", escape(arg), ps.stdin); + } + } + + // Standard streams + if (spec.getStdInputLocation().overlaps(FileLocation.MEMORY)) { + throw new IllegalArgumentException("In-memory STDIN not supported"); + } + + if (spec.getStdInput() != null) { + writeSpec("stdin.location", spec.getStdInputLocation().toString(), ps.stdin); + writeSpec("stdin.path", spec.getStdInput(), ps.stdin); + } + + writeStandardFileSpec(spec.getStdOutput(), spec.getStdOutputLocation(), "stdout", ps); + writeStandardFileSpec(spec.getStdError(), spec.getStdErrorLocation(), "stderr", ps); + + // Environment variables + for (String en : spec.getEnvironmentVariableNames()) { + writeSpec("env." + en, escape(spec.getEnvironmentVariable(en)), ps.stdin); + } + + // Attributes + for (String attr : spec.getAttributeNames()) { + Object o = spec.getAttribute(attr); + if (o != null) { + writeSpec("attr." + attr, escape(o.toString()), ps.stdin); + } + } + + // Staging stuff + writeStagingSpec(spec.getStageIn(), "stagein", false, ps); + writeStagingSpec(spec.getStageOut(), "stageout", true, ps); + + // Cleanup + if (spec.getCleanUpSet() != null) { + for (String ce : spec.getCleanUpSet()) { + writeSpec("cleanup", escape(ce), ps.stdin); + } + } + + ps.stdin.close(); + + try { + int ec = p.waitFor(); + if (logger.isDebugEnabled()) { + logger.debug(props.getSubmitCommandName() + " done (exit code " + ec + ")"); + } + if (ec != 0) { + throw new ProcessException(props.getSubmitCommand() + " failed: " + read(ps.stderr)); + } + } + catch (InterruptedException e) { + throw new ProcessException(e); + } + + parseOutput(ps.stdout); + active = true; + if (logger.isDebugEnabled()) { + logger.debug("Submitted job with id '" + jobid + "'"); + } + getQueuePoller().addJob( + job = createJob(jobid, stdout, spec.getStdOutputLocation(), stderr, + spec.getStdErrorLocation(), exitcode, this)); + } + + + private void parseOutput(Reader r) throws IOException, ProcessException { + BufferedReader br = new BufferedReader(r); + String line; + while ((line = br.readLine()) != null) { + String[] els = line.split("=", 2); + if (els.length < 2) { + throw new ProcessException("Invalid output from '" + + props.getSubmitCommand() + "': " + line); + } + if ("jobid".equals(els[0])) { + jobid = els[1]; + if ("".equals(jobid)) { + throw new ProcessException("Received empty job id from '" + + props.getSubmitCommand() + "'"); + } + } + else if ("stdout.path".equals(els[0])) { + stdout = els[1]; + } + else if ("stderr.path".equals(els[0])) { + stderr = els[1]; + } + else { + throw new ProcessException("Invalid output from '" + + props.getSubmitCommand() + "': " + line); + } + } + if (jobid == null) { + throw new ProcessException("No job id received from '" + props.getSubmitCommand() + "'"); + } + } + + @Override + public void processCompleted(int exitCode) { + active = false; + super.processCompleted(exitCode); + } + + + @Override + public void processFailed(String message) { + active = false; + super.processFailed(message); + } + + + @Override + public void processFailed(Exception e) { + active = false; + super.processFailed(e); + } + + + public void cancel() throws TaskSubmissionException { + if (!active) { + throw new TaskSubmissionException("Can only cancel an active task"); + } + String[] cmdline = new String[] { getProperties().getRemoveCommand(), jobid }; + try { + logger.debug("Canceling job: jobid=" + jobid); + Process process = Runtime.getRuntime().exec(cmdline, null, null); + int ec = process.waitFor(); + if (ec != 0) { + throw new TaskSubmissionException("Failed to cancel task: " + + read(new InputStreamReader(process.getInputStream()))); + } + } + catch (InterruptedException e) { + throw new TaskSubmissionException( + "Thread interrupted while waiting for " + + getProperties().getRemoveCommandName() + " to finish"); + } + catch (IOException e) { + throw new TaskSubmissionException("Failed to cancel task", e); + } + } + + private String escape(String value) { + if (value == null) { + return null; + } + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < value.length(); i++) { + char c = value.charAt(i); + if (c == '\n' || c == '\\') { + sb.append("\\"); + if (c == '\n') { + sb.append('n'); + } + else { + sb.append('\\'); + } + } + else { + sb.append(c); + } + } + return sb.toString(); + } + + private String read(Reader r) throws IOException { + StringBuilder sb = new StringBuilder(); + BufferedReader br = new BufferedReader(r); + String line = br.readLine(); + while (line != null) { + sb.append(line); + sb.append('\n'); + line = br.readLine(); + } + return sb.toString(); + } + + private void writeStagingSpec(StagingSet ss, String name, boolean modes, ProcessStreams pc) throws IOException { + if (ss == null || ss.isEmpty()) { + return; + } + for (StagingSetEntry se : ss) { + writeSpec(name + ".source", escape(se.getSource()), pc.stdin); + writeSpec(name + ".destination", escape(se.getDestination()), pc.stdin); + if (modes) { + writeSpec(name + ".mode", stagingModeToStr(se.getMode()), pc.stdin); + } + } + } + + private String stagingModeToStr(EnumSet mode) { + int s = 0; + for (Mode m : mode) { + s += m.getId(); + } + return String.valueOf(s); + } + + private void writeStandardFileSpec(String path, FileLocation loc, String name, ProcessStreams pc) + throws IOException { + if (path == null) { + if (loc.overlaps(FileLocation.MEMORY)) { + writeSpec(name + ".location", "tmp", pc.stdin); + } + } + else { + writeSpec("stdout.location", loc.remove(FileLocation.MEMORY).toString(), pc.stdin); + writeSpec("stdout.path", path, pc.stdin); + } + } + + @Override + protected void writeScript(Writer wr, String exitcode, String stdout, String stderr) + throws IOException { + // not used + } + + @Override + protected String getName() { + return "Shell"; + } + + @Override + protected AbstractProperties getProperties() { + return props; + } + + @Override + protected Job createJob(String jobid, String stdout, + FileLocation stdOutputLocation, String stderr, + FileLocation stdErrorLocation, String exitcode, + AbstractExecutor executor) { + return new Job(jobid, stdout, stdOutputLocation, stderr, + stdErrorLocation, exitcode, executor); + } + + private static QueuePoller poller; + + @Override + protected AbstractQueuePoller getQueuePoller() { + synchronized(ShellExecutor.class) { + if (poller == null) { + poller = new QueuePoller(props); + poller.start(); + } + return poller; + } + } +} Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/Properties.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/Properties.java (revision 0) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/shell/Properties.java (revision 3918) @@ -0,0 +1,54 @@ +//---------------------------------------------------------------------- +//This code is developed as part of the Java CoG Kit project +//The terms of the license can be found at http://www.cogkit.org/license +//This message may not be removed or altered. +//---------------------------------------------------------------------- + +/* + * Created on Oct 20, 2005 + */ +package org.globus.cog.abstraction.impl.scheduler.shell; + +// import org.apache.log4j.Logger; +import org.globus.cog.abstraction.impl.scheduler.common.AbstractProperties; + +public class Properties extends AbstractProperties { + + private static final long serialVersionUID = 1L; + + public static final String POLL_INTERVAL = "poll.interval"; + public static final String SUBMIT_COMMAND = "submit.command"; + public static final String POLL_COMMAND = "poll.command"; + public static final String CANCEL_COMMAND = "cancel.command"; + public static final String NAME = "name"; + + public static synchronized Properties getProperties(String name) { + Properties properties = new Properties(); + properties.load("provider-" + name + ".properties"); + properties.put(NAME, name); + return properties; + } + + protected void setDefaults() { + setPollInterval(5); + } + + + public String getPollCommandName() { + return POLL_COMMAND; + } + + + public String getRemoveCommandName() { + return CANCEL_COMMAND; + } + + + public String getSubmitCommandName() { + return SUBMIT_COMMAND; + } + + public String getName() { + return getProperty(NAME); + } +} Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/slurm/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/slurm/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/slurm/execution/TaskHandlerImpl.java (working copy) @@ -15,7 +15,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "Slurm"; } } \ No newline at end of file Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/cobalt/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/cobalt/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/cobalt/execution/TaskHandlerImpl.java (working copy) @@ -22,12 +22,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "Cobalt"; } - - public String toString() - { - return "TaskHandlerImpl(execution cobalt)"; - } } Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/condor/execution/TaskHandlerImpl.java (working copy) @@ -22,7 +22,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "Condor"; } } Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/execution/TaskHandlerImpl.java (working copy) @@ -21,12 +21,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "LSF"; } - - public String toString() - { - return "TaskHandlerImpl(execution LSF)"; - } } Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/execution/TaskHandlerImpl.java (working copy) @@ -22,7 +22,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "PBS"; } } \ No newline at end of file Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (working copy) @@ -61,11 +61,10 @@ validate(task); } - public void start() - throws AuthorizationException, IOException, ProcessException { + public void start() throws AuthorizationException, IOException, ProcessException { String scriptdirPath = System.getProperty("script.dir"); - if(scriptdirPath == null) { + if (scriptdirPath == null) { scriptdirPath = System.getProperty("user.home") + File.separatorChar + ".globus" + File.separatorChar + "scripts"; @@ -107,6 +106,10 @@ try { int code = process.waitFor(); + if (logger.isDebugEnabled()) { + logger.debug(getProperties().getSubmitCommandName() + + " done (exit code " + code + ")"); + } if (code != 0) { String errorText = getOutput(process.getInputStream()) + getOutput(process.getErrorStream()); @@ -115,10 +118,6 @@ + " reported an exit code of " + code + "). " + errorText); } - if (logger.isDebugEnabled()) { - logger.debug(getProperties().getSubmitCommandName() - + " done (exit code " + code + ")"); - } } catch (InterruptedException e) { if (logger.isDebugEnabled()) { @@ -126,8 +125,7 @@ + getProperties().getSubmitCommandName(), e); } if (listener != null) { - listener - .processFailed("The submission process was interrupted"); + listener.processFailed("The submission process was interrupted"); } } @@ -268,8 +266,7 @@ } protected abstract void writeScript(Writer wr, String exitcode, - String stdout, String stderr) - throws IOException; + String stdout, String stderr) throws IOException; protected JobSpecification getSpec() { return spec; @@ -403,8 +400,7 @@ @param list May be null or empty @throws IOException */ - protected void writeQuotedList(Writer writer, List list) - throws IOException + protected void writeQuotedList(Writer writer, List list) throws IOException { if (list != null && list.size() > 0) { writer.write(' '); Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractQueuePoller.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractQueuePoller.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractQueuePoller.java (working copy) @@ -15,7 +15,6 @@ import java.io.InputStreamReader; import java.util.Arrays; import java.util.HashMap; -import java.util.Iterator; import java.util.LinkedList; import java.util.Map; @@ -28,7 +27,8 @@ public static final int MAX_CONSECUTIVE_FAILURES = 3; private String name; - private LinkedList newjobs, donejobs; + private LinkedList newjobs; + private LinkedList donejobs; private Map jobs; boolean any = false; private int sleepTime; @@ -45,8 +45,8 @@ this.properties = properties; this.sleepTime = properties.getPollInterval() * 1000; jobs = new HashMap(); - newjobs = new LinkedList(); - donejobs = new LinkedList(); + newjobs = new LinkedList(); + donejobs = new LinkedList(); } public void start() { @@ -102,7 +102,7 @@ else { synchronized (newjobs) { while (!newjobs.isEmpty()) { - Job job = (Job) newjobs.removeFirst(); + Job job = newjobs.removeFirst(); jobs.put(job.getJobID(), job); } } @@ -115,7 +115,7 @@ } else { while (!donejobs.isEmpty()) { - String jobid = (String) donejobs.removeFirst(); + String jobid = donejobs.removeFirst(); removeDoneJob(jobid); } } @@ -137,9 +137,7 @@ if (logger.isDebugEnabled()) { logger.debug("Fail all: " + message); } - Iterator i = jobs.values().iterator(); - while (i.hasNext()) { - Job job = (Job) i.next(); + for (Job job : jobs.values()) { try { job.fail(message); } Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/Job.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/Job.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/Job.java (working copy) @@ -24,11 +24,12 @@ public static final int STATE_QUEUED = 1; public static final int STATE_RUNNING = 2; public static final int STATE_DONE = 3; + public static final int STATE_FAILED = 5; public static final int STATE_UNKNOWN = 4; private static final int NO_EXITCODE = -1; - private String jobID, location; + private String jobID, location, message; private String exitcodeFileName; private String stdout, stderr; private FileLocation outLoc, errLoc; @@ -50,8 +51,12 @@ this.ticks = 0; this.exitcode = NO_EXITCODE; } + + public void setMessage(String message) { + this.message = message; + } - public boolean close() { + public boolean close(int tentativeState) { if (logger.isDebugEnabled()) { logger.debug("Closing " + jobID); } @@ -63,9 +68,8 @@ f = new File(exitcodeFileName); if (f != null && !f.exists()) { if (ticks == 5) { - listener - .processFailed(new ProcessException( - "Exitcode file (" + exitcodeFileName + ") not found 5 queue polls after the job was reported done")); + listener.processFailed(new ProcessException( + "Exitcode file (" + exitcodeFileName + ") not found 5 queue polls after the job was reported done")); return true; } else { @@ -78,11 +82,11 @@ } } - processExitCode(); + processExitCode(tentativeState); return true; } - protected boolean processExitCode() { + protected boolean processExitCode(int tentativeState) { if (logger.isDebugEnabled()) { logger.debug("Processing exit code for job " + jobID); } @@ -101,7 +105,17 @@ if (logger.isDebugEnabled()) { logger.debug("Exit code: " + exitcode); } - listener.processCompleted(exitcode); + if (tentativeState == STATE_FAILED) { + if (message == null) { + listener.processFailed("Job failed (exit code: " + exitcode + ")"); + } + else { + listener.processFailed(message); + } + } + else { + listener.processCompleted(exitcode); + } return true; } catch (Exception e) { @@ -155,7 +169,7 @@ } public synchronized void await() { - while (state != STATE_DONE) { + while (state != STATE_DONE && state != STATE_FAILED) { try { wait(); } @@ -173,9 +187,8 @@ return; } else { - if (state == STATE_DONE) { - if (close()) { - this.state = STATE_DONE; + if (state == STATE_DONE || state == STATE_FAILED) { + if (close(state)) { synchronized (this) { notify(); } Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractJobSubmissionTaskHandler.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractJobSubmissionTaskHandler.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractJobSubmissionTaskHandler.java (working copy) @@ -14,7 +14,6 @@ import org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException; import org.globus.cog.abstraction.impl.common.task.InvalidServiceContactException; import org.globus.cog.abstraction.impl.common.task.TaskSubmissionException; -import org.globus.cog.abstraction.interfaces.DelegatedTaskHandler; import org.globus.cog.abstraction.interfaces.FileLocation; import org.globus.cog.abstraction.interfaces.JobSpecification; import org.globus.cog.abstraction.interfaces.Status; Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/execution/TaskHandlerImpl.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/execution/TaskHandlerImpl.java (revision 3917) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/execution/TaskHandlerImpl.java (working copy) @@ -22,7 +22,7 @@ return new JobSubmissionTaskHandler(); } - protected String getName() { + public String getName() { return "SGE"; } } \ No newline at end of file From davidk at ci.uchicago.edu Tue Jun 10 11:51:16 2014 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Tue, 10 Jun 2014 11:51:16 -0500 (CDT) Subject: [Swift-commit] r7911 - in www: downloads inc Message-ID: <20140610165116.1CFA89D5A1@svn.ci.uchicago.edu> Author: davidk Date: 2014-06-10 11:51:15 -0500 (Tue, 10 Jun 2014) New Revision: 7911 Modified: www/downloads/index.php www/inc/downloads_sidebar.php Log: A few updates from last week Modified: www/downloads/index.php =================================================================== --- www/downloads/index.php 2014-06-09 22:37:36 UTC (rev 7910) +++ www/downloads/index.php 2014-06-10 16:51:15 UTC (rev 7911) @@ -59,14 +59,22 @@ Precompiled binary distribution [swift-0.94.1.tar.gz] [Release notes] +>swift-0.94.1.tar.gz] [Release notes] +

      Beta Version

      +

      +

      Swift 0.95 RC6

      +This is a beta for the upcoming Swift 0.95 release. +
      +[swift-0.95-RC6.tar.gz]

      Source Code

      The source code for Swift is available to developers who have an interest in contributing -new features. The code is +new features. The code is browsable online, or accessible through Subversion. To download and compile the source code, you will need Apache Ant and Java JDK. Modified: www/inc/downloads_sidebar.php =================================================================== --- www/inc/downloads_sidebar.php 2014-06-09 22:37:36 UTC (rev 7910) +++ www/inc/downloads_sidebar.php 2014-06-10 16:51:15 UTC (rev 7911) @@ -1,7 +1,7 @@

      -
      Download Now
      -0.94.1 RC3 current version
      2013/09/04 +
      Download Now
      +0.94.1 current version
      2013/09/04

       

      From ketan at ci.uchicago.edu Thu Jun 12 12:23:21 2014 From: ketan at ci.uchicago.edu (ketan at ci.uchicago.edu) Date: Thu, 12 Jun 2014 12:23:21 -0500 (CDT) Subject: [Swift-commit] r7912 - in SwiftApps/swift-galaxy/swift: . powergrid Message-ID: <20140612172321.E1ABD9DA62@svn.ci.uchicago.edu> Author: ketan Date: 2014-06-12 12:23:20 -0500 (Thu, 12 Jun 2014) New Revision: 7912 Added: SwiftApps/swift-galaxy/swift/powergrid/ SwiftApps/swift-galaxy/swift/powergrid/ampl.xml SwiftApps/swift-galaxy/swift/powergrid/characterization.png SwiftApps/swift-galaxy/swift/powergrid/gensample.xml SwiftApps/swift-galaxy/swift/powergrid/lower.xml SwiftApps/swift-galaxy/swift/powergrid/powergrid.xml SwiftApps/swift-galaxy/swift/powergrid/runmatlab.xml SwiftApps/swift-galaxy/swift/powergrid/swift-inference.xml SwiftApps/swift-galaxy/swift/powergrid/swift-lmps SwiftApps/swift-galaxy/swift/powergrid/swift-lmps.xml SwiftApps/swift-galaxy/swift/powergrid/swift-powergrid SwiftApps/swift-galaxy/swift/powergrid/upper.xml Log: powergrid galaxy tools Added: SwiftApps/swift-galaxy/swift/powergrid/ampl.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/ampl.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/ampl.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,13 @@ + + Run the inference model to calculate bounds + + + run_ampl.py "${sample}" "${fix_x}" "${what}" "${stoch_model}" "${xhat_out}" "${obj_out}" "${illinois}" "${edrevised_model}" "${ascii_data}" "${print_results}" "{ampl_res}" + + + + + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/characterization.png =================================================================== (Binary files differ) Property changes on: SwiftApps/swift-galaxy/swift/powergrid/characterization.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftApps/swift-galaxy/swift/powergrid/gensample.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/gensample.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/gensample.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,12 @@ + + Generate a random sample + + /home/ubuntu/powergridapps/bin/gensample.py "${sample}" "${outfile}" + + + + + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/lower.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/lower.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/lower.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,3 @@ + + Run the inference model to calculate lower bound + Added: SwiftApps/swift-galaxy/swift/powergrid/powergrid.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/powergrid.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/powergrid.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,32 @@ + + Swift - Power Grid Inference for Stochastic Optimization + + + swift-powergrid $site $batch $log_file + + + + + + + + + + + + + + + + + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/runmatlab.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/runmatlab.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/runmatlab.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,12 @@ + + Generate a random sample + + gensample.py "${sample}" "${outfile}" + + + + + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/swift-inference.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/swift-inference.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/swift-inference.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,55 @@ + + Swift - Power Grid Inference for Stochastic Optimization + + + swift-powergrid $site $batch $sample $log_file + + + + + + + + + + + + + + + + + + +**The Swift power grid inference analysis tool. This tool runs an inference model written in AMPL using the Coin IPOPT solver.** + +----- + +The electrical power prices in a region are a result of +combination of many stochastic and temporal factors, including variation in +supply and demand due to market, social, and environmental factors. Evaluating +the feasibility of future generation power grid networks and renewable energy +sources requires modeling and simulation of this complex system. In particular, +the power grid application described here is used to statistically infer the +changes in the unit commitment prices with respect to small variations in +random factors. + +----- + +The application involves running a stochastic model for a +large number of elements generated via a three-level nested *foreach* +loop. A numerical algorithm is run to compute lower and upper bounds, which +converge for large enough samples. A moderate sample size of five samples can +generate hundreds of thousands of tasks. Each task makes call to the +Python-implemented sample generation and AMPL models. + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/swift-lmps =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/swift-lmps (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/swift-lmps 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,276 @@ +#!/bin/bash + +site=$1 +numhrs=$2 +state=$3 +#edrevised=$4 +#asciidata=$5 +#printres=$6 +#illinois=$7 +#wind_data=$8 +#sample=$3 +logfile=$4 + +#cp $stochmodel stochmodel.run +#cp $edrevised edrevised.mod +#cp $asciidata set_data_ascii.inc +#cp $printres print_results_revised.inc +#cp $illinois Illinois.tgz +#cp $wind_data wind_data.dat +echo $@>/tmp/tmp.txt +#swift=/home/maheshwari/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift +swift=$(\which swift) +# use "here" document for configs + +wdir=$(mktemp -d /tmp/swift-gal.XXXX) + +cat << EOF > $wdir/lmps4galaxy.swift +type file; + +app (file _amplout, file _stdout, file _stderr) runampl (file _model, file _bundle, int _idx){ + amplrun @_model @_bundle _idx @_amplout stdout=@_stdout stderr=@_stderr; +} + +app (file _plot, file _stdout, file _stderr) runmatlab (file _data, file _bundle){ + matlabrun @_data @_bundle @_plot stdout=@_stdout stderr=@_stderr; +} + +/* +for i in `seq 1 8760`; do echo $i>idx_data.dat; ampl edrevised.run > /sandbox/ketan/std.out.$i; mv Results/lmp_results.dat /sandbox/ketan/lmp_results.$i.dat; done +*/ + +file model<"/home/ubuntu/powergridapps/KETAN_VERSION/MODEL_LMP/edrevised.run">; +file bundle<"/home/ubuntu/powergridapps/KETAN_VERSION/MODEL_LMP/bundle.tgz">; + +file amplout[] ; +file out[] ; +file err[] ; + +/* +file plotres[] ; +file plotout[] ; +file ploterr[] ; +*/ + +foreach i in [1:${numhrs}]{ + (amplout[i], out[i], err[i]) = runampl (model, bundle, i); + //(plotres[i], plotout[i], ploterr[i]) = runmatlab (amplout[i]); +} +EOF + +case "$site" in +midway ) +cat << EOF > $wdir/sites.xml + + + + + sandyb + 16 + 02:00:00 + 7500 + 100 + 100 + 2 + 1 + 1 + .31 + 10000 + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +midway echo echo +midway cat cat +midway gensample /home/maheshwari/powergridapps/bin/gensample.py +midway runampl /home/maheshwari/powergridapps/bin/run_ampl.py +midway runampl_l /home/maheshwari/powergridapps/bin/run_ampl_l.py +midway runampl_u /home/maheshwari/powergridapps/bin/run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +use.wrapper.staging=false +EOF +;; + +uc3 ) +cat << EOF > $wdir/sites.xml + + + + + .09 + 1000 + 1 + 3600 + 1 + 100 + 100 + 1 + UidDomain == "osg-gk.mwt2.org" + 1 + "group_friends.{env.USER}" + nonshared + + + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +uc3 echo /bin/echo +uc3 cat /bin/cat +uc3 runampl run_ampl.py +uc3 runampl_l run_ampl_l.py +uc3 runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=1 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +EOF +;; + +stampede ) +cat << EOF > $wdir/sites.xml + + + + + + 16 + 16 + TG-EAR130015 + normal + ketancmaheshwari at gmail.com + ALL + 02:00:00 + 8000 + 100 + 100 + 1 + 1 + 1 + .15 + 10000 + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +stampede echo echo +stampede cat cat +stampede gensample gensample.py +stampede runampl run_ampl.py +stampede runampl_l run_ampl_l.py +stampede runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +use.wrapper.staging=false +EOF +;; + +localhost ) +cat << EOF > $wdir/sites.xml + + + + + /tmp/swift.workdir + 0.1 + + +EOF + +cat << EOF > $wdir/tc +localhost echo echo +localhost cat cat +localhost amplrun runampl.sh +localhost gensample gensample.py +localhost runampl run_ampl.py +localhost runampl_l run_ampl_l.py +localhost runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=false +lazy.errors=false +foreach.max.threads=32 +EOF +;; + +* ) +echo "defaulting to localhost" +cat << EOF > $wdir/sites.xml + + + + + /tmp/swift.workdir + 0.1 + + +EOF + +cat << EOF > $wdir/tc +localhost echo echo +localhost cat cat +localhost gensample gensample.py +localhost runampl run_ampl.py +localhost runampl_l run_ampl_l.py +localhost runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=false +lazy.errors=false +foreach.max.threads=32 +EOF +;; +esac + +cd $wdir + +$swift -ui http:9090 -sites.file sites.xml -tc.file tc -config cf lmps4galaxy.swift 2> std.err +#$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/inference4galaxy.swift -batch=$batch -stochmodel=stochmodel.run -edrevised=edrevised.mod -asciidata=set_data_ascii.inc -printres=print_results_revised.inc -illinois=Illinois.tgz -wind_data=wind_data.dat 2> std.err + +EXITCODE=$? + +if [ "$EXITCODE" -ne "0" ]; then + cat std.err >&2 +fi +cp std.err $logfile +rm std.err +exit $EXITCODE + +cd .. + Property changes on: SwiftApps/swift-galaxy/swift/powergrid/swift-lmps ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/swift-galaxy/swift/powergrid/swift-lmps.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/swift-lmps.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/swift-lmps.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,57 @@ + + Swift - Power Grid LMP Computation + + + swift-lmps $site $numhrs $state $log_file + + + + + + + + + + + + + + + + + + + + + + + + +**The Swift power grid locational marginal price computation tool. This tool runs an inference model written in AMPL using the Coin IPOPT solver.** +----- +Optimal power flow studies are crucial in understanding the flow and price +patterns in electricity under different demand and network conditions. A big +computational challenge arising in power grid analysis is that simulations need +to be run at high time resolutions in order to capture effects occurring at +multiple time scales. The power flow simulation application analyzes historical +conditions in the Illinois grid to simulate instant power prices on an hourly +basis. + +----- +The application runs linear programming solvers invoked via an AMPL (A +Mathematical Programming Language) model representation and collects flow, +generation, and price data with attached geographical coordinates. A typical +application consists of running the model in 8,760 independent executions +corresponding to each hour of the year. + + + + Added: SwiftApps/swift-galaxy/swift/powergrid/swift-powergrid =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/swift-powergrid (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/swift-powergrid 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,311 @@ +#!/bin/bash + +site=$1 +batch=$2 +#stochmodel=$3 +#edrevised=$4 +#asciidata=$5 +#printres=$6 +#illinois=$7 +#wind_data=$8 +sample=$3 +logfile=$4 + +#cp $stochmodel stochmodel.run +#cp $edrevised edrevised.mod +#cp $asciidata set_data_ascii.inc +#cp $printres print_results_revised.inc +#cp $illinois Illinois.tgz +#cp $wind_data wind_data.dat +echo $@>>/tmp/tmp.txt +#swift=/home/maheshwari/swift-0.94/cog/modules/swift/dist/swift-svn/bin/swift +swift=$(\which swift) +# use "here" document for configs + +wdir=$(mktemp -d /tmp/swift-gal.XXXX) + +cat << EOF > $wdir/inference4galaxy.swift +type file; + +#string basepath="/home/ubuntu/powergridapps/swiftscripts"; + +file stoch_model<"/home/ubuntu/powergridapps/swiftscripts/stochmodel.mod">; +file edrevised_model<"/home/ubuntu/powergridapps/swiftscripts/edrevised.mod">; +file ascii_data<"/home/ubuntu/powergridapps/swiftscripts/set_data_ascii.inc">; +file print_results<"/home/ubuntu/powergridapps/swiftscripts/print_results_revised.inc">; +file illinois<"/home/ubuntu/powergridapps/swiftscripts/Illinois.tgz">; +file wind_data<"/home/ubuntu/powergridapps/swiftscripts/wind_data.dat">; + +file norm_out[] ; + +file out[] ; +file obj_out[] ; +file ampl_res[] ; + +file xhat_out[] ; + +file out_l[][][] ; +file obj_out_l[][][] ; +file ampl_l_res[][][] ; + +file out_u[][][] ; +file obj_out_u[][][] ; +file ampl_u_res[][][] ; + +app ( file _norm_out ) gensample (int _s, file _wind_data){ + + gensample _s @_norm_out @_wind_data; +} + +app ( file _obj_out, file _xhat_out, file _ampl_res, file _out) + ampl_app ( int _s, int _fix_x, file _what, file _stoch_model, file _illinois, file _edrevised_model, file _ascii_data, file _print_results) { + + runampl _s _fix_x @_what @_stoch_model @_xhat_out @_obj_out @_illinois @_edrevised_model @_ascii_data @_print_results @_ampl_res stdout=@_out; +} + +app (file _obj_out, file _ampl_l_res, file _out) ampl_app_L ( + int _s, int _fix_x, file _what, file _stoch_model, file _illinois, file _edrevised_model, file _ascii_data, file _print_results) { + + runampl_l _s _fix_x @_what @_stoch_model @_obj_out @_illinois @_edrevised_model @_ascii_data @_print_results @_ampl_l_res stdout=@_out; +} + +app (file _obj_out, file _ampl_u_res, file _out) ampl_app_U ( + int _s, int _fix_x, file _what, file _stoch_model, file _xhat, file _illinois, file _edrevised_model, file _ascii_data, file _print_results) { + + runampl_u _s _fix_x @_what @_stoch_model @_xhat @_obj_out @_illinois @_edrevised_model @_ascii_data @_print_results @_ampl_u_res stdout=@_out; +} + +//int nS[] = [10, 100]; //number of scenario sweeps +int nS[] = [${sample}]; //number of scenario sweeps + +int Batch = @toInt(@arg("batch", "30")); + +foreach S, idxs in nS{ + (norm_out[idxs]) = gensample (S, wind_data); + (obj_out[idxs], xhat_out[idxs], ampl_res[idxs], out[idxs]) = ampl_app (S, 0, norm_out[idxs], stoch_model, illinois, edrevised_model, ascii_data, print_results); + + foreach B, idxb in [10:Batch:10]{ + foreach k in [0:B]{ + (obj_out_l[idxs][idxb][k], ampl_l_res[idxs][idxb][k], out_l[idxs][idxb][k]) = ampl_app_L(S, 0, norm_out[idxs], stoch_model, illinois, edrevised_model, ascii_data, print_results); + (obj_out_u[idxs][idxb][k], ampl_u_res[idxs][idxb][k], out_u[idxs][idxb][k]) = ampl_app_U(S, 1, norm_out[idxs], stoch_model, xhat_out[idxs], illinois, edrevised_model, ascii_data, print_results); + } + } +} + +EOF + +case "$site" in +midway ) +cat << EOF > $wdir/sites.xml + + + + + sandyb + 16 + 02:00:00 + 7500 + 100 + 100 + 2 + 1 + 1 + .31 + 10000 + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +midway echo echo +midway cat cat +midway gensample /home/maheshwari/powergridapps/bin/gensample.py +midway runampl /home/maheshwari/powergridapps/bin/run_ampl.py +midway runampl_l /home/maheshwari/powergridapps/bin/run_ampl_l.py +midway runampl_u /home/maheshwari/powergridapps/bin/run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +use.wrapper.staging=false +EOF +;; + +uc3 ) +cat << EOF > $wdir/sites.xml + + + + + .09 + 1000 + 1 + 3600 + 1 + 100 + 100 + 1 + UidDomain == "osg-gk.mwt2.org" + 1 + "group_friends.{env.USER}" + nonshared + + + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +uc3 echo /bin/echo +uc3 cat /bin/cat +uc3 runampl run_ampl.py +uc3 runampl_l run_ampl_l.py +uc3 runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=1 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +EOF +;; + +stampede ) +cat << EOF > $wdir/sites.xml + + + + + + 16 + 16 + TG-EAR130015 + normal + ketancmaheshwari at gmail.com + ALL + 02:00:00 + 8000 + 100 + 100 + 1 + 1 + 1 + .15 + 10000 + /tmp/swift.work + + +EOF + +cat << EOF > $wdir/tc +stampede echo echo +stampede cat cat +stampede gensample gensample.py +stampede runampl run_ampl.py +stampede runampl_l run_ampl_l.py +stampede runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=true +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=true +lazy.errors=false +use.wrapper.staging=false +EOF +;; + +localhost ) +cat << EOF > $wdir/sites.xml + + + + + /tmp/swift.workdir + 0.1 + + +EOF + +cat << EOF > $wdir/tc +localhost echo echo +localhost cat cat +localhost gensample gensample.py +localhost runampl run_ampl.py +localhost runampl_l run_ampl_l.py +localhost runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=false +lazy.errors=false +foreach.max.threads=32 +EOF +;; + +* ) +echo "defaulting to localhost" +cat << EOF > $wdir/sites.xml + + + + + /tmp/swift.workdir + 0.1 + + +EOF + +cat << EOF > $wdir/tc +localhost echo echo +localhost cat cat +localhost gensample gensample.py +localhost runampl run_ampl.py +localhost runampl_l run_ampl_l.py +localhost runampl_u run_ampl_u.py +EOF + +cat << EOF > $wdir/cf +use.provider.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +provider.staging.pin.swiftfiles=false +sitedir.keep=false +lazy.errors=false +foreach.max.threads=32 +EOF +;; +esac + +cd $wdir + +$swift -ui http:9090 -sites.file sites.xml -tc.file tc -config cf inference4galaxy.swift -batch=$batch 2> std.err +#$swift -sites.file /tmp/sites.xml -tc.file /tmp/tc -config /tmp/cf /tmp/inference4galaxy.swift -batch=$batch -stochmodel=stochmodel.run -edrevised=edrevised.mod -asciidata=set_data_ascii.inc -printres=print_results_revised.inc -illinois=Illinois.tgz -wind_data=wind_data.dat 2> std.err + +EXITCODE=$? + +if [ "$EXITCODE" -ne "0" ]; then + cat std.err >&2 +fi +cp std.err $logfile +rm std.err +exit $EXITCODE + +cd .. + Property changes on: SwiftApps/swift-galaxy/swift/powergrid/swift-powergrid ___________________________________________________________________ Added: svn:executable + * Added: SwiftApps/swift-galaxy/swift/powergrid/upper.xml =================================================================== --- SwiftApps/swift-galaxy/swift/powergrid/upper.xml (rev 0) +++ SwiftApps/swift-galaxy/swift/powergrid/upper.xml 2014-06-12 17:23:20 UTC (rev 7912) @@ -0,0 +1,3 @@ + + Run the inference model to calculate upper bound + From hategan at ci.uchicago.edu Thu Jun 12 20:36:19 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Thu, 12 Jun 2014 20:36:19 -0500 (CDT) Subject: [Swift-commit] r7913 - trunk/src/org/griphyn/vdl/util Message-ID: <20140613013619.768BC9D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-12 20:36:19 -0500 (Thu, 12 Jun 2014) New Revision: 7913 Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java Log: only check config properties after all the files have been loaded Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-12 17:23:20 UTC (rev 7912) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 01:36:19 UTC (rev 7913) @@ -79,7 +79,9 @@ } c.load(file); config = c; - return config.check(); + config.check(); + config.validateProperties(); + return config; } private List files, tried; @@ -153,7 +155,13 @@ } } - protected VDL2Config check() throws IOException { + private void validateProperties() { + for (Map.Entry e : this.entrySet()) { + checkType(e.getKey(), e.getValue()); + } + } + + protected VDL2Config check() throws IOException { if (files.size() == 0) { throw new FileNotFoundException("No Swift configuration file found. Tried " + tried); } @@ -175,7 +183,6 @@ public synchronized Object put(Object key, Object value) { String svalue = (String) value; if (svalue.indexOf("${") == -1) { - checkType(key, value); return super.put(key, value); } else { @@ -209,7 +216,6 @@ } sb.append(svalue.substring(last)); value = sb.toString(); - checkType(key, value); return super.put(key, value); } } From hategan at ci.uchicago.edu Thu Jun 12 20:38:21 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Thu, 12 Jun 2014 20:38:21 -0500 (CDT) Subject: [Swift-commit] r7914 - branches/release-0.95/src/org/griphyn/vdl/util Message-ID: <20140613013821.808D59D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-12 20:38:21 -0500 (Thu, 12 Jun 2014) New Revision: 7914 Modified: branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java Log: only check properties after all files have been loaded Modified: branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 01:36:19 UTC (rev 7913) +++ branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 01:38:21 UTC (rev 7914) @@ -79,7 +79,9 @@ } c.load(file); config = c; - return config.check(); + config.check(); + config.validateProperties(); + return config; } private List files, tried; @@ -153,7 +155,13 @@ } } - protected VDL2Config check() throws IOException { + private void validateProperties() { + for (Map.Entry e : this.entrySet()) { + checkType(e.getKey(), e.getValue()); + } + } + + protected VDL2Config check() throws IOException { if (files.size() == 0) { throw new FileNotFoundException("No Swift configuration file found. Tried " + tried); } @@ -175,7 +183,6 @@ public synchronized Object put(Object key, Object value) { String svalue = (String) value; if (svalue.indexOf("${") == -1) { - checkType(key, value); return super.put(key, value); } else { @@ -209,7 +216,6 @@ } sb.append(svalue.substring(last)); value = sb.toString(); - checkType(key, value); return super.put(key, value); } } From hategan at ci.uchicago.edu Thu Jun 12 21:05:17 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Thu, 12 Jun 2014 21:05:17 -0500 (CDT) Subject: [Swift-commit] r7915 - in trunk/src/org/griphyn/vdl: karajan util Message-ID: <20140613020517.B2B539D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-12 21:05:17 -0500 (Thu, 12 Jun 2014) New Revision: 7915 Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java trunk/src/org/griphyn/vdl/util/ConfigPropertyType.java trunk/src/org/griphyn/vdl/util/VDL2Config.java Log: better property validation scheme where command line overrides are taken into account and the file in which the problem is found is printed with the error Modified: trunk/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/Loader.java 2014-06-13 01:38:21 UTC (rev 7914) +++ trunk/src/org/griphyn/vdl/karajan/Loader.java 2014-06-13 02:05:17 UTC (rev 7915) @@ -126,9 +126,10 @@ source = null; } - setupLogging(ap, projectName, runID); VDL2Config config = loadConfig(ap); addCommandLineProperties(config, ap); + config.validateProperties(); + setupLogging(ap, projectName, runID); logBasicInfo(argv, runID, config); debugSitesText(config); debugTCText(config); @@ -537,14 +538,12 @@ private static void addCommandLineProperties(VDL2Config config, ArgumentParser ap) { + config.setCurrentFile(""); Map desc = VDL2ConfigProperties.getPropertyDescriptions(); for (Map.Entry e : desc.entrySet()) { String name = e.getKey(); if (ap.isPresent(name)) { String value = ap.getStringValue(name); - if (logger.isDebugEnabled()) { - logger.debug("setting: " + name + " to: " + value); - } config.setProperty(name, value); } } Modified: trunk/src/org/griphyn/vdl/util/ConfigPropertyType.java =================================================================== --- trunk/src/org/griphyn/vdl/util/ConfigPropertyType.java 2014-06-13 01:38:21 UTC (rev 7914) +++ trunk/src/org/griphyn/vdl/util/ConfigPropertyType.java 2014-06-13 02:05:17 UTC (rev 7915) @@ -27,7 +27,7 @@ return new Choices(values); } - public abstract void checkValue(String propName, String value); + public abstract void checkValue(String propName, String value, String source); private static String pp(Collection c) { StringBuilder sb = new StringBuilder(); @@ -54,9 +54,9 @@ } @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { if (!choices.contains(value)) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Valid values are: " + pp(choices)); } } @@ -64,19 +64,19 @@ private static class CPTString extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { // all values accepted } } private static class Int extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { try { Integer.parseInt(value); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Must be an integer"); } } @@ -84,12 +84,12 @@ private static class CPTFloat extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { try { Double.parseDouble(value); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Must be a floating point number."); } } @@ -97,10 +97,10 @@ private static class CPTFile extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { File f = new File(value); if (!f.exists()) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. File does not exist."); } } Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 01:38:21 UTC (rev 7914) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 02:05:17 UTC (rev 7915) @@ -80,16 +80,18 @@ c.load(file); config = c; config.check(); - config.validateProperties(); return config; } private List files, tried; private Map types; + private Map propertySource; + private String currentFile; private VDL2Config() { files = new LinkedList(); tried = new LinkedList(); + propertySource = new HashMap(); types = new HashMap(); put(VDL2ConfigProperties.POOL_FILE, "${swift.home}/etc/sites.xml", ConfigPropertyType.FILE); put(VDL2ConfigProperties.TC_FILE, "${swift.home}/etc/tc.data", ConfigPropertyType.FILE); @@ -132,9 +134,11 @@ private VDL2Config(VDL2Config other) { this.putAll(other); this.files.addAll(other.files); + this.propertySource.putAll(other.propertySource); } protected void load(String file) throws IOException { + this.currentFile = file; tried.add(file); File f = new File(file); if (f.exists()) { @@ -155,7 +159,7 @@ } } - private void validateProperties() { + public void validateProperties() { for (Map.Entry e : this.entrySet()) { checkType(e.getKey(), e.getValue()); } @@ -181,6 +185,7 @@ * occur. */ public synchronized Object put(Object key, Object value) { + propertySource.put(key, currentFile); String svalue = (String) value; if (svalue.indexOf("${") == -1) { return super.put(key, value); @@ -223,7 +228,7 @@ private void checkType(Object key, Object value) { ConfigPropertyType type = types.get(key); if (type != null) { - type.checkValue((String) key, (String) value); + type.checkValue((String) key, (String) value, propertySource.get(key)); } } @@ -287,6 +292,8 @@ public Object clone() { VDL2Config conf = new VDL2Config(); conf.putAll(this); + conf.files.addAll(files); + conf.propertySource.putAll(propertySource); return conf; } @@ -305,4 +312,8 @@ } return getProperty(name); } + + public void setCurrentFile(String f) { + this.currentFile = f; + } } From hategan at ci.uchicago.edu Thu Jun 12 21:10:23 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Thu, 12 Jun 2014 21:10:23 -0500 (CDT) Subject: [Swift-commit] r7916 - in branches/release-0.95/src: . org/griphyn/vdl/karajan org/griphyn/vdl/util Message-ID: <20140613021023.AE5B09D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-12 21:10:23 -0500 (Thu, 12 Jun 2014) New Revision: 7916 Modified: branches/release-0.95/src/ branches/release-0.95/src/org/griphyn/vdl/karajan/Loader.java branches/release-0.95/src/org/griphyn/vdl/util/ConfigPropertyType.java branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java Log: port of properties fix take II from trunk Property changes on: branches/release-0.95/src ___________________________________________________________________ Modified: svn:mergeinfo - /branches/release-0.94/src:6283-6619,6999-7001,7125 /trunk/src:6214,6255,6275-6618 + /branches/release-0.94/src:6283-6619,6999-7001,7125 /trunk/src:6214,6255,6275-6618,7913,7915 Modified: branches/release-0.95/src/org/griphyn/vdl/karajan/Loader.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/karajan/Loader.java 2014-06-13 02:05:17 UTC (rev 7915) +++ branches/release-0.95/src/org/griphyn/vdl/karajan/Loader.java 2014-06-13 02:10:23 UTC (rev 7916) @@ -110,9 +110,10 @@ checkValidProject(project); String projectName = projectName(project); - setupLogging(ap, projectName, runID); VDL2Config config = loadConfig(ap); addCommandLineProperties(config, ap); + config.validateProperties(); + setupLogging(ap, projectName, runID); logBasicInfo(argv, runID, config); debugSitesText(config); debugTCText(config); @@ -472,14 +473,12 @@ private static void addCommandLineProperties(VDL2Config config, ArgumentParser ap) { + config.setCurrentFile(""); Map desc = VDL2ConfigProperties.getPropertyDescriptions(); for (Map.Entry e : desc.entrySet()) { String name = e.getKey(); if (ap.isPresent(name)) { String value = ap.getStringValue(name); - if (logger.isDebugEnabled()) { - logger.debug("setting: " + name + " to: " + value); - } config.setProperty(name, value); } } Modified: branches/release-0.95/src/org/griphyn/vdl/util/ConfigPropertyType.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/util/ConfigPropertyType.java 2014-06-13 02:05:17 UTC (rev 7915) +++ branches/release-0.95/src/org/griphyn/vdl/util/ConfigPropertyType.java 2014-06-13 02:10:23 UTC (rev 7916) @@ -27,7 +27,7 @@ return new Choices(values); } - public abstract void checkValue(String propName, String value); + public abstract void checkValue(String propName, String value, String source); private static String pp(Collection c) { StringBuilder sb = new StringBuilder(); @@ -54,9 +54,9 @@ } @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { if (!choices.contains(value)) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Valid values are: " + pp(choices)); } } @@ -64,19 +64,19 @@ private static class CPTString extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { // all values accepted } } private static class Int extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { try { Integer.parseInt(value); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Must be an integer"); } } @@ -84,12 +84,12 @@ private static class CPTFloat extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { try { Double.parseDouble(value); } catch (NumberFormatException e) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. Must be a floating point number."); } } @@ -97,10 +97,10 @@ private static class CPTFile extends ConfigPropertyType { @Override - public void checkValue(String propName, String value) { + public void checkValue(String propName, String value, String source) { File f = new File(value); if (!f.exists()) { - throw new IllegalArgumentException("Invalid value '" + value + "' for property '" + + throw new IllegalArgumentException(source + ":\n\tInvalid value '" + value + "' for property '" + propName + "'. File does not exist."); } } Modified: branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 02:05:17 UTC (rev 7915) +++ branches/release-0.95/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-13 02:10:23 UTC (rev 7916) @@ -80,16 +80,18 @@ c.load(file); config = c; config.check(); - config.validateProperties(); return config; } private List files, tried; private Map types; + private Map propertySource; + private String currentFile; private VDL2Config() { files = new LinkedList(); tried = new LinkedList(); + propertySource = new HashMap(); types = new HashMap(); put(VDL2ConfigProperties.POOL_FILE, "${swift.home}/etc/sites.xml", ConfigPropertyType.FILE); put(VDL2ConfigProperties.TC_FILE, "${swift.home}/etc/tc.data", ConfigPropertyType.FILE); @@ -132,9 +134,11 @@ private VDL2Config(VDL2Config other) { this.putAll(other); this.files.addAll(other.files); + this.propertySource.putAll(other.propertySource); } protected void load(String file) throws IOException { + this.currentFile = file; tried.add(file); File f = new File(file); if (f.exists()) { @@ -155,7 +159,7 @@ } } - private void validateProperties() { + public void validateProperties() { for (Map.Entry e : this.entrySet()) { checkType(e.getKey(), e.getValue()); } @@ -181,6 +185,7 @@ * occur. */ public synchronized Object put(Object key, Object value) { + propertySource.put(key, currentFile); String svalue = (String) value; if (svalue.indexOf("${") == -1) { return super.put(key, value); @@ -223,7 +228,7 @@ private void checkType(Object key, Object value) { ConfigPropertyType type = types.get(key); if (type != null) { - type.checkValue((String) key, (String) value); + type.checkValue((String) key, (String) value, propertySource.get(key)); } } @@ -287,6 +292,8 @@ public Object clone() { VDL2Config conf = new VDL2Config(); conf.putAll(this); + conf.files.addAll(files); + conf.propertySource.putAll(propertySource); return conf; } @@ -305,4 +312,8 @@ } return getProperty(name); } + + public void setCurrentFile(String f) { + this.currentFile = f; + } } From yadunandb at ci.uchicago.edu Fri Jun 13 11:08:52 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Fri, 13 Jun 2014 11:08:52 -0500 (CDT) Subject: [Swift-commit] r7917 - trunk/tests/multi_remote Message-ID: <20140613160852.7E1809D04F@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-13 11:08:51 -0500 (Fri, 13 Jun 2014) New Revision: 7917 Modified: trunk/tests/multi_remote/remote_driver.check.sh trunk/tests/multi_remote/remote_driver.setup.sh Log: Updates to publish server Modified: trunk/tests/multi_remote/remote_driver.check.sh =================================================================== --- trunk/tests/multi_remote/remote_driver.check.sh 2014-06-13 02:10:23 UTC (rev 7916) +++ trunk/tests/multi_remote/remote_driver.check.sh 2014-06-13 16:08:51 UTC (rev 7917) @@ -1,7 +1,9 @@ #!/bin/bash PUBLISH_LOCATION=/scratch/midway/yadunand/publish -PUBLISH_SERVER=http://swift.rcc.uchicago.edu:8042 +#PUBLISH_SERVER=http://swift.rcc.uchicago.edu:8042 +PUBLISH_SERVER=http://users.rcc.uchicago.edu/~yadunand + DATE=$(date +%H%M%S) [ ! -z $FROM_MAIL ] || FROM_MAIL="test-engine at swift.rcc.uchicago" Modified: trunk/tests/multi_remote/remote_driver.setup.sh =================================================================== --- trunk/tests/multi_remote/remote_driver.setup.sh 2014-06-13 02:10:23 UTC (rev 7916) +++ trunk/tests/multi_remote/remote_driver.setup.sh 2014-06-13 16:08:51 UTC (rev 7917) @@ -187,7 +187,7 @@ echo "Folder cleaned" PID=$$ -CPIDS=$(pgrep -P $PID); (sleep 20 && kill -KILL $CPIDS &); kill -TERM $CPIDS; +CPIDS=$(pgrep -P $PID); (sleep 20 && kill -SIGKILL $CPIDS &); kill -TERM $CPIDS; exit 0 EOF From yadunandb at ci.uchicago.edu Fri Jun 13 11:09:26 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Fri, 13 Jun 2014 11:09:26 -0500 (CDT) Subject: [Swift-commit] r7918 - in branches/release-0.95/tests: multi_remote stress/apps/modis_uc3 Message-ID: <20140613160926.619C39D04F@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-13 11:09:26 -0500 (Fri, 13 Jun 2014) New Revision: 7918 Modified: branches/release-0.95/tests/multi_remote/remote_driver.check.sh branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml Log: Updates to publish server Updated URL for osgconnect Modified: branches/release-0.95/tests/multi_remote/remote_driver.check.sh =================================================================== --- branches/release-0.95/tests/multi_remote/remote_driver.check.sh 2014-06-13 16:08:51 UTC (rev 7917) +++ branches/release-0.95/tests/multi_remote/remote_driver.check.sh 2014-06-13 16:09:26 UTC (rev 7918) @@ -1,7 +1,8 @@ #!/bin/bash PUBLISH_LOCATION=/scratch/midway/yadunand/publish -PUBLISH_SERVER=http://swift.rcc.uchicago.edu:8042 +#PUBLISH_SERVER=http://swift.rcc.uchicago.edu:8042 +PUBLISH_SERVER=http://users.rcc.uchicago.edu/~yadunand DATE=$(date +%H%M%S) [ ! -z $FROM_MAIL ] || FROM_MAIL="test-engine at swift.rcc.uchicago" Modified: branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-13 16:08:51 UTC (rev 7917) +++ branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-13 16:09:26 UTC (rev 7918) @@ -2,7 +2,7 @@ - + 10.00 10000 1 @@ -16,7 +16,7 @@ "group_friends.{env.USER}" "Swift" nonshared - . + /home/{env.USER}/swiftwork From yadunandb at ci.uchicago.edu Fri Jun 13 11:22:07 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Fri, 13 Jun 2014 11:22:07 -0500 (CDT) Subject: [Swift-commit] r7919 - branches/release-0.95/tests/sites/local-coasters Message-ID: <20140613162207.B5B439D04F@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-13 11:22:07 -0500 (Fri, 13 Jun 2014) New Revision: 7919 Modified: branches/release-0.95/tests/sites/local-coasters/sites.template.xml Log: Updated sites.file for local-coasters test Modified: branches/release-0.95/tests/sites/local-coasters/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/local-coasters/sites.template.xml 2014-06-13 16:09:26 UTC (rev 7918) +++ branches/release-0.95/tests/sites/local-coasters/sites.template.xml 2014-06-13 16:22:07 UTC (rev 7919) @@ -1,16 +1,9 @@ - + + - - - - _WORK_ - file - - - - _HOST_ + 2.55 10000 4 From yadunandb at ci.uchicago.edu Fri Jun 13 15:28:34 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Fri, 13 Jun 2014 15:28:34 -0500 (CDT) Subject: [Swift-commit] r7920 - branches/release-0.95/tests/stress/random_fail Message-ID: <20140613202834.5183B9D6CE@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-13 15:28:34 -0500 (Fri, 13 Jun 2014) New Revision: 7920 Added: branches/release-0.95/tests/stress/random_fail/randfail.sh Modified: branches/release-0.95/tests/stress/random_fail/sites.template.xml branches/release-0.95/tests/stress/random_fail/tc.template.data Log: Updated randfail regression test Added: branches/release-0.95/tests/stress/random_fail/randfail.sh =================================================================== --- branches/release-0.95/tests/stress/random_fail/randfail.sh (rev 0) +++ branches/release-0.95/tests/stress/random_fail/randfail.sh 2014-06-13 20:28:34 UTC (rev 7920) @@ -0,0 +1,15 @@ +#!/bin/bash + +FAIL_PROBABILITY=$1 +DELAY=$2 + +ITEM=$(($RANDOM%100)) +sleep $2 + +if (( "$ITEM" <= "$FAIL_PROBABILITY" )) +then + echo "Failing $ITEM < $FAIL_PROBABILITY" >&2 + exit -1 +fi +echo "Not failing $ITEM > $FAIL_PROBABILITY" +exit 0 Modified: branches/release-0.95/tests/stress/random_fail/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/random_fail/sites.template.xml 2014-06-13 16:22:07 UTC (rev 7919) +++ branches/release-0.95/tests/stress/random_fail/sites.template.xml 2014-06-13 20:28:34 UTC (rev 7920) @@ -1,20 +1,13 @@ - - - sandyb - 16 - 36000 - 00:10:00 - 100 - 100 - 4 - 1 - 1 - .64 - 10000 - /tmp/MIDWAY_USERNAME + + + + 10000 + .31 + file + /tmp/{env.USER}/swiftwork Modified: branches/release-0.95/tests/stress/random_fail/tc.template.data =================================================================== --- branches/release-0.95/tests/stress/random_fail/tc.template.data 2014-06-13 16:22:07 UTC (rev 7919) +++ branches/release-0.95/tests/stress/random_fail/tc.template.data 2014-06-13 20:28:34 UTC (rev 7920) @@ -1,2 +1 @@ -local perl /usr/bin/perl null null null -local sh /bin/bash +localhost sh /bin/bash null null null From hategan at ci.uchicago.edu Fri Jun 13 21:54:53 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 13 Jun 2014 21:54:53 -0500 (CDT) Subject: [Swift-commit] r7921 - branches/release-0.95/src/org/griphyn/vdl/karajan Message-ID: <20140614025453.5B5AA9D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-13 21:54:53 -0500 (Fri, 13 Jun 2014) New Revision: 7921 Modified: branches/release-0.95/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java Log: fixed npe when app not in tc.data and some apps are in running state Modified: branches/release-0.95/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java =================================================================== --- branches/release-0.95/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2014-06-13 20:28:34 UTC (rev 7920) +++ branches/release-0.95/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2014-06-14 02:54:53 UTC (rev 7921) @@ -41,6 +41,7 @@ import org.globus.cog.abstraction.interfaces.StatusListener; import org.globus.cog.abstraction.interfaces.Task; import org.globus.cog.karajan.scheduler.AbstractScheduler; +import org.globus.cog.karajan.scheduler.ContactAllocationTask; import org.globus.cog.karajan.scheduler.ResourceConstraintChecker; import org.globus.cog.karajan.scheduler.TaskConstraints; import org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler; @@ -396,8 +397,10 @@ } Task t = e.task; List cluster = null; - synchronized (tasks) { - cluster = tasks.get(t); + if (!(t instanceof ContactAllocationTask)) { + synchronized (tasks) { + cluster = tasks.get(t); + } } if (cluster != null) { for (Entry e1 : cluster) { From hategan at ci.uchicago.edu Fri Jun 13 21:57:18 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Fri, 13 Jun 2014 21:57:18 -0500 (CDT) Subject: [Swift-commit] r7922 - trunk/src/org/griphyn/vdl/karajan Message-ID: <20140614025718.B14579D6CE@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-13 21:57:18 -0500 (Fri, 13 Jun 2014) New Revision: 7922 Modified: trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java Log: port of NPE missing tc.data app fix from 0.95 Modified: trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java =================================================================== --- trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2014-06-14 02:54:53 UTC (rev 7921) +++ trunk/src/org/griphyn/vdl/karajan/VDSAdaptiveScheduler.java 2014-06-14 02:57:18 UTC (rev 7922) @@ -41,6 +41,7 @@ import org.globus.cog.abstraction.interfaces.StatusListener; import org.globus.cog.abstraction.interfaces.Task; import org.globus.cog.karajan.scheduler.AbstractScheduler; +import org.globus.cog.karajan.scheduler.ContactAllocationTask; import org.globus.cog.karajan.scheduler.ResourceConstraintChecker; import org.globus.cog.karajan.scheduler.TaskConstraints; import org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler; @@ -396,8 +397,10 @@ } Task t = e.task; List cluster = null; - synchronized (tasks) { - cluster = tasks.get(t); + if (!(t instanceof ContactAllocationTask)) { + synchronized (tasks) { + cluster = tasks.get(t); + } } if (cluster != null) { for (Entry e1 : cluster) { From yadunandb at ci.uchicago.edu Sat Jun 14 16:07:39 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Sat, 14 Jun 2014 16:07:39 -0500 (CDT) Subject: [Swift-commit] r7923 - in branches/release-0.95/tests: . groups random_fail sites/beagle sites/local-coasters sites/mcs sites/osgconnect sites/raven stress stress/IO/uc3 stress/apps/modis_local stress/apps/modis_midway stress/apps/modis_uc3 stress/local_cluster Message-ID: <20140614210739.20F9B9D036@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-14 16:07:38 -0500 (Sat, 14 Jun 2014) New Revision: 7923 Added: branches/release-0.95/tests/random_fail/ branches/release-0.95/tests/random_fail/Bug_info branches/release-0.95/tests/random_fail/rand_fail_Bug1067.check.sh branches/release-0.95/tests/random_fail/rand_fail_Bug1067.setup.sh branches/release-0.95/tests/random_fail/rand_fail_Bug1067.swift branches/release-0.95/tests/random_fail/run branches/release-0.95/tests/random_fail/sites.template.xml branches/release-0.95/tests/random_fail/swift.properties branches/release-0.95/tests/random_fail/tc.template.data branches/release-0.95/tests/random_fail/title.txt branches/release-0.95/tests/sites/local-coasters/sites.backup branches/release-0.95/tests/sites/raven/sites.template.backup branches/release-0.95/tests/stress/local_cluster/ branches/release-0.95/tests/stress/local_cluster/combiner.sh branches/release-0.95/tests/stress/local_cluster/run branches/release-0.95/tests/stress/local_cluster/simple_MapRed.args branches/release-0.95/tests/stress/local_cluster/simple_MapRed.check.sh branches/release-0.95/tests/stress/local_cluster/simple_MapRed.setup.sh branches/release-0.95/tests/stress/local_cluster/simple_MapRed.stdout branches/release-0.95/tests/stress/local_cluster/simple_MapRed.swift branches/release-0.95/tests/stress/local_cluster/simple_MapRed.timeout branches/release-0.95/tests/stress/local_cluster/sites.template.xml branches/release-0.95/tests/stress/local_cluster/sites.xml branches/release-0.95/tests/stress/local_cluster/swift.properties branches/release-0.95/tests/stress/local_cluster/teragen_wrap.sh branches/release-0.95/tests/stress/local_cluster/title.txt Removed: branches/release-0.95/tests/sites/beagle/sanity.setup.sh branches/release-0.95/tests/sites/beagle/sanity.source.sh branches/release-0.95/tests/sites/beagle/sanity.swift branches/release-0.95/tests/sites/beagle/sanity.timeout branches/release-0.95/tests/sites/mcs/sanity.setup.sh branches/release-0.95/tests/sites/mcs/sanity.source.sh branches/release-0.95/tests/sites/mcs/sanity.swift branches/release-0.95/tests/sites/mcs/sanity.timeout Modified: branches/release-0.95/tests/groups/group-daily-remote.sh branches/release-0.95/tests/sites/beagle/sites.template.xml branches/release-0.95/tests/sites/osgconnect/sites.template.xml branches/release-0.95/tests/sites/raven/sites.template.xml branches/release-0.95/tests/stress/IO/uc3/sites.template.xml branches/release-0.95/tests/stress/apps/modis_local/sites.template.xml branches/release-0.95/tests/stress/apps/modis_midway/sites.template.xml branches/release-0.95/tests/stress/apps/modis_uc3/modis.timeout branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml Log: Fixes for test-suite. FIxed modis, local_cluster, random_fail and several tests failing due to remote site quotas being hit and due to lack of housekeeping. I should probably add a housekeeping job to the tail end of the test suite. Modified: branches/release-0.95/tests/groups/group-daily-remote.sh =================================================================== --- branches/release-0.95/tests/groups/group-daily-remote.sh 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/groups/group-daily-remote.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -25,15 +25,19 @@ $TESTDIR/functions \ # Site testing test-group + $TESTDIR/sites/local \ + $TESTDIR/sites/local-coasters \ + $TESTDIR/sites/multiple_coaster_pools \ + $TESTDIR/sites/beagle \ $TESTDIR/sites/mcs \ $TESTDIR/sites/midway \ - $TESTDIR/sites/uc3 \ + $TESTDIR/sites/osgconnect \ # Frisbee will fail due to Bug 1030 $TESTDIR/sites/mac-frisbee \ $TESTDIR/sites/blues \ $TESTDIR/sites/fusion \ - #$TESTDIR/sites/raven \ + $TESTDIR/sites/raven \ # Remote-cluster IO tests $TESTDIR/stress/IO/beagle \ Added: branches/release-0.95/tests/random_fail/Bug_info =================================================================== --- branches/release-0.95/tests/random_fail/Bug_info (rev 0) +++ branches/release-0.95/tests/random_fail/Bug_info 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,22 @@ +Regression tests for Bug 1067 +| Link -> http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1067 +Swift 0.94 swift-r6888 cog-r3762 + +Exception in sh: + Arguments: [randfail.sh, 50, 0] + Host: local + Directory: rand_fail_Bug1067-20130820-1749-6n7zbux4/jobs/6/sh-6abm13el + stderr.txt: Failing 11 < 50 + stdout.txt: +Caused by: Application /bin/bash failed with an exit code of 255 + +Exception in sh: + Arguments: [randfail.sh, 50, 0] + Host: local + Directory: rand_fail_Bug1067-20130820-1749-6n7zbux4/jobs/8/sh-8abm13el + stderr.txt: Failing 19 < 50 + stdout.txt: +Caused by: Application /bin/bash failed with an exit code of 255 + +Execution failed: + Got one name (derr) and 0 values: [] \ No newline at end of file Added: branches/release-0.95/tests/random_fail/rand_fail_Bug1067.check.sh =================================================================== --- branches/release-0.95/tests/random_fail/rand_fail_Bug1067.check.sh (rev 0) +++ branches/release-0.95/tests/random_fail/rand_fail_Bug1067.check.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,17 @@ +#!/bin/bash + +if [ ! -f ${0%.check.sh}.stdout ] +then + echo "${$0%.check.sh}.stdout missing" + exit -1 +fi + +grep "Got one name (derr)" ${0%.check.sh}.stdout +if [ "$?" == 0 ] +then + echo "EXIT : REGRESSION FOUND!" >&2 + exit -1 +else + echo "Test passed" +fi +exit 0 \ No newline at end of file Property changes on: branches/release-0.95/tests/random_fail/rand_fail_Bug1067.check.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/random_fail/rand_fail_Bug1067.setup.sh =================================================================== --- branches/release-0.95/tests/random_fail/rand_fail_Bug1067.setup.sh (rev 0) +++ branches/release-0.95/tests/random_fail/rand_fail_Bug1067.setup.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,22 @@ +#!/bin/bash + +# Setup script will just output the following file + +cat<<'EOF' > randfail.sh +#!/bin/bash + +FAIL_PROBABILITY=$1 +DELAY=$2 + +ITEM=$(($RANDOM%100)) +sleep $2 + +if (( "$ITEM" <= "$FAIL_PROBABILITY" )) +then + echo "Failing $ITEM < $FAIL_PROBABILITY" >&2 + exit -1 +fi +echo "Not failing $ITEM > $FAIL_PROBABILITY" +exit 0 +EOF + Property changes on: branches/release-0.95/tests/random_fail/rand_fail_Bug1067.setup.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/random_fail/rand_fail_Bug1067.swift =================================================================== --- branches/release-0.95/tests/random_fail/rand_fail_Bug1067.swift (rev 0) +++ branches/release-0.95/tests/random_fail/rand_fail_Bug1067.swift 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,34 @@ +type file; + +file script<"randfail.sh">; + +app (file ofile1, file ofile2) quicklyFailingApp(file script, int failchance, +int delay) +{ + sh @script failchance delay stdout=@ofile1 stderr=@ofile2; +} + +app (file ofile) someApp3(file ifile, file jfile, file kfile) +{ + sh "-c" @strcat("cat ", at filename(ifile)) stdout=@ofile; +} + +app (file ofile) someApp(file ifile) +{ + sh "-c" @strcat("cat ", at filename(ifile)) stdout=@ofile; +} + +app sleep (int sec) +{ + sh "-c" @strcat("sleep ",sec); +} + +int sufficientlyLargeNumber = 100; + +file a[]; +foreach i in [0:sufficientlyLargeNumber] { + file f1; + file f2; + (f1,f2) = quicklyFailingApp(script,50,0); + a[i] = someApp(f2); +} Added: branches/release-0.95/tests/random_fail/run =================================================================== --- branches/release-0.95/tests/random_fail/run (rev 0) +++ branches/release-0.95/tests/random_fail/run 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,21 @@ +#!/bin/bash + +PATH=/scratch/midway/yadunand/swift-0.94RC2/cog/modules/swift/dist/swift-svn/bin:$PATH + + +echo "Swift location: "; which swift +echo "Swift version : "; swift -version + +rm rand_fail_Bug1067.stdout +cat title.txt + +./rand_fail_Bug1067.setup.sh + +for i in `seq 1 10` +do +swift -tc.file tc.template.data -config swift.properties -sites.file sites.template.xml rand_fail_Bug1067.swift | tee -a rand_fail_Bug1067.stdout + +rm -rf *{swiftx,kml} rand_fail_Bug1067-* _concurrent* failed* +done + +./rand_fail_Bug1067.check.sh \ No newline at end of file Property changes on: branches/release-0.95/tests/random_fail/run ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/random_fail/sites.template.xml =================================================================== --- branches/release-0.95/tests/random_fail/sites.template.xml (rev 0) +++ branches/release-0.95/tests/random_fail/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,20 @@ + + + + + + sandyb + 16 + 36000 + 00:10:00 + 100 + 100 + 4 + 1 + 1 + .64 + 10000 + /tmp/MIDWAY_USERNAME + + + Added: branches/release-0.95/tests/random_fail/swift.properties =================================================================== --- branches/release-0.95/tests/random_fail/swift.properties (rev 0) +++ branches/release-0.95/tests/random_fail/swift.properties 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,11 @@ +wrapperlog.always.transfer=true +sitedir.keep=true +file.gc.enabled=false +status.mode=provider + +execution.retries=5 +lazy.errors=true + +use.wrapper.staging=false +use.provider.staging=false +provider.staging.pin.swiftfiles=falsewrapperlog.always.transfer=true Added: branches/release-0.95/tests/random_fail/tc.template.data =================================================================== --- branches/release-0.95/tests/random_fail/tc.template.data (rev 0) +++ branches/release-0.95/tests/random_fail/tc.template.data 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,2 @@ +local perl /usr/bin/perl null null null +local sh /bin/bash Added: branches/release-0.95/tests/random_fail/title.txt =================================================================== --- branches/release-0.95/tests/random_fail/title.txt (rev 0) +++ branches/release-0.95/tests/random_fail/title.txt 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,2 @@ +Regression tests for Bug 1067 +| Link -> http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1067 Deleted: branches/release-0.95/tests/sites/beagle/sanity.setup.sh =================================================================== --- branches/release-0.95/tests/sites/beagle/sanity.setup.sh 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/beagle/sanity.setup.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,15 +0,0 @@ -#!/bin/bash - -echo "BEAGLE_USERNAME : $BEAGLE_USERNAME" -echo "MIDWAY_USERNAME : $MIDWAY_USERNAME" -echo "MCS_USERNAME : $MCS_USERNAME" -echo "UC3_USERNAME : $UC3_USERNAME" -USERNAME=$BEAGLE_USERNAME - -if [[ -z $USERNAME ]] -then - echo "Remote username not provided. Skipping sites configs" -else - ls *xml - cat sites.xml | sed "s/{env.USER}/$USERNAME/" > tmp && mv tmp sites.xml -fi Deleted: branches/release-0.95/tests/sites/beagle/sanity.source.sh =================================================================== --- branches/release-0.95/tests/sites/beagle/sanity.source.sh 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/beagle/sanity.source.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,6 +0,0 @@ -#!/bin/bash -if [ "$HOSTNAME" == "midway001" ] -then - export GLOBUS_HOSTNAME=swift.rcc.uchicago.edu - export GLOBUS_TCP_PORT_RANGE=50000,51000 -fi; Deleted: branches/release-0.95/tests/sites/beagle/sanity.swift =================================================================== --- branches/release-0.95/tests/sites/beagle/sanity.swift 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/beagle/sanity.swift 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,11 +0,0 @@ -type file; - -app (file out, file err) remote_driver () -{ - date stdout=@filename(out) stderr=@filename(err); -} - -file driver_out ; -file driver_err ; - -(driver_out, driver_err) = remote_driver(); \ No newline at end of file Deleted: branches/release-0.95/tests/sites/beagle/sanity.timeout =================================================================== --- branches/release-0.95/tests/sites/beagle/sanity.timeout 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/beagle/sanity.timeout 2014-06-14 21:07:38 UTC (rev 7923) @@ -1 +0,0 @@ -300 Modified: branches/release-0.95/tests/sites/beagle/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/beagle/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/beagle/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -19,4 +19,4 @@ /tmp/{env.USER}/swiftwork - \ No newline at end of file + Added: branches/release-0.95/tests/sites/local-coasters/sites.backup =================================================================== --- branches/release-0.95/tests/sites/local-coasters/sites.backup (rev 0) +++ branches/release-0.95/tests/sites/local-coasters/sites.backup 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,24 @@ + + + + + + _WORK_ + file + + + + + + + 2.55 + 10000 + 4 + 8 + 1000 + 1 + 4 + _WORK_ + + + Deleted: branches/release-0.95/tests/sites/mcs/sanity.setup.sh =================================================================== --- branches/release-0.95/tests/sites/mcs/sanity.setup.sh 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/mcs/sanity.setup.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,11 +0,0 @@ -#!/bin/bash - -USERNAME=$MCS_USERNAME - -if [[ -z $USERNAME ]] -then - echo "Remote username not provided. Skipping sites configs" -else - ls *xml - cat sites.xml | sed "s/{env.USER}/$USERNAME/" > tmp && mv tmp sites.xml -fi Deleted: branches/release-0.95/tests/sites/mcs/sanity.source.sh =================================================================== --- branches/release-0.95/tests/sites/mcs/sanity.source.sh 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/mcs/sanity.source.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,6 +0,0 @@ -#!/bin/bash -if [ "$HOSTNAME" == "midway001" ] -then - export GLOBUS_HOSTNAME=swift.rcc.uchicago.edu - export GLOBUS_TCP_PORT_RANGE=50000,51000 -fi; Deleted: branches/release-0.95/tests/sites/mcs/sanity.swift =================================================================== --- branches/release-0.95/tests/sites/mcs/sanity.swift 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/mcs/sanity.swift 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,11 +0,0 @@ -type file; - -app (file out, file err) remote_driver () -{ - date stdout=@filename(out) stderr=@filename(err); -} - -file driver_out ; -file driver_err ; - -(driver_out, driver_err) = remote_driver(); \ No newline at end of file Deleted: branches/release-0.95/tests/sites/mcs/sanity.timeout =================================================================== --- branches/release-0.95/tests/sites/mcs/sanity.timeout 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/mcs/sanity.timeout 2014-06-14 21:07:38 UTC (rev 7923) @@ -1 +0,0 @@ -300 Modified: branches/release-0.95/tests/sites/osgconnect/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/osgconnect/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/osgconnect/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -2,7 +2,7 @@ - + 10.00 10000 1 Added: branches/release-0.95/tests/sites/raven/sites.template.backup =================================================================== --- branches/release-0.95/tests/sites/raven/sites.template.backup (rev 0) +++ branches/release-0.95/tests/sites/raven/sites.template.backup 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,21 @@ + + + + + + CI-SES000031 + + 24 + pbs.aprun;pbs.mpp;depth=24 + 00:01:00 + 1 + 1 + 1 + 1.00 + 10000 + + /home/users/{env.USER}/swiftwork + + + + Modified: branches/release-0.95/tests/sites/raven/sites.template.xml =================================================================== --- branches/release-0.95/tests/sites/raven/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/sites/raven/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,9 +1,9 @@ + CI-SES000031 - KEEP 24 pbs.aprun;pbs.mpp;depth=24 00:01:00 @@ -13,7 +13,8 @@ 1.00 10000 - /home/users/{env.USER}/swiftwork + /home/users/p01898/swiftwork/ + Modified: branches/release-0.95/tests/stress/IO/uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/IO/uc3/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/stress/IO/uc3/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -2,7 +2,7 @@ - + 10.00 10000 1 Modified: branches/release-0.95/tests/stress/apps/modis_local/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/apps/modis_local/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/stress/apps/modis_local/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -1,5 +1,16 @@ + + + + + 10000 + .31 + _WORK_ + + + + Modified: branches/release-0.95/tests/stress/apps/modis_midway/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/apps/modis_midway/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/stress/apps/modis_midway/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -4,16 +4,16 @@ westmere - 16 00:05:00 3600 100 100 - 2 + 1 1 + 8 1 .64 10000 - /tmp/{env.USER} + /scratch/midway/{env.USER}/swiftwork Modified: branches/release-0.95/tests/stress/apps/modis_uc3/modis.timeout =================================================================== --- branches/release-0.95/tests/stress/apps/modis_uc3/modis.timeout 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/stress/apps/modis_uc3/modis.timeout 2014-06-14 21:07:38 UTC (rev 7923) @@ -1 +1 @@ -600 +900 Modified: branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-14 02:57:18 UTC (rev 7922) +++ branches/release-0.95/tests/stress/apps/modis_uc3/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -3,20 +3,19 @@ - 10.00 - 10000 - 1 3600 - 00:10:00 - 100 - 100 - 1 + 00:30:00 + + 4 1 + 4 1 - "group_friends.{env.USER}" - "Swift" + 10.00 + 10000 + nonshared - /home/{env.USER}/swiftwork - + "Swift" + . + Added: branches/release-0.95/tests/stress/local_cluster/combiner.sh =================================================================== --- branches/release-0.95/tests/stress/local_cluster/combiner.sh (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/combiner.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,16 @@ +#!/bin/bash + +FILES=$* +SUM=0 +COUNT=0 + +for file in $* +do + RES=($(awk '{ sum += $1 } END { print sum,NR }' $file)) + echo "${RES[0]} ${RES[1]}" + SUM=$(($SUM+${RES[0]})) + COUNT=$(($COUNT+${RES[1]})) +done +echo "SUM : $SUM" +echo "COUNT: $COUNT" +exit 0 Property changes on: branches/release-0.95/tests/stress/local_cluster/combiner.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/local_cluster/run =================================================================== --- branches/release-0.95/tests/stress/local_cluster/run (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/run 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,27 @@ +#!/bin/bash + +PATH=/scratch/midway/yadunand/swift-0.94RC2/cog/modules/swift/dist/swift-svn/bin:$PATH + +echo "Swift location: "; which swift +echo "Swift version : "; swift -version + +export MIDWAY_USERNAME=yadunand +export BEAGLE_USERNAME=yadunandb +export MCS_USERNAME=yadunand +export UC3_USERNAME=yadunand + +SCRIPT=simple_MapRed.swift +BASE=${SCRIPT%.swift} + +rm $BASE.stdout +cat title.txt + +cp sites.template.xml sites.xml +./$BASE.setup.sh + +ARGS=$(cat $BASE.args) +swift -tc.file tc.data -config swift.properties -sites.file sites.xml $BASE.swift ${ARGS[*]} | tee -a $BASE.stdout + +rm -rf *{swiftx,kml} $BASE-* _concurrent* failed* &> /dev/null + +./$BASE.check.sh \ No newline at end of file Property changes on: branches/release-0.95/tests/stress/local_cluster/run ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.args =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.args (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.args 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1 @@ +-loops=10 \ No newline at end of file Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.check.sh =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.check.sh (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.check.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,21 @@ +#!/bin/bash + +BASE=${0%.check.sh} +ARGS=`cat $BASE.args | sed 's/-loops=//'` + +EXPECTED=$(($ARGS * 10000)) + +if [ -f "final_result" ];then + RESULT=($(tail -n 1 final_result)) + echo "RESULT line : ${RESULT[*]}" + echo "EXPECTED = $EXPECTED" + echo "ACTUAL = ${RESULT[1]}" +fi + +if [[ "${RESULT[1]}" == "$EXPECTED" ]] +then + echo "Result matched" +else + echo "Result does not match expectation" >&2 + exit 1 +fi Property changes on: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.check.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.setup.sh =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.setup.sh (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.setup.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,86 @@ +#!/bin/bash + +HOST=$(hostname -f) + +if [[ "$HOST" == *midway* ]]; then + echo "On Midway" + echo "midway bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *beagle* ]]; then + echo "On Beagle" + echo "beagle bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *mcs* ]]; then + echo "On MCS" + echo "mcs bash /bin/bash null null null" > tc.data +elif [[ "$HOST" == *uc3* ]]; then + echo "On UC3" + echo "uc3 bash /bin/bash null null null" > tc.data +else + echo "On unidentified machine, using defaults" + echo "local bash /bin/bash null null null" > tc.data +fi + +if [[ -z $MIDWAY_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/{mid.USER}/$MIDWAY_USERNAME/" > tmp && mv tmp\ + sites.xml +fi +if [[ -z $UC3_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/{uc3.USER}/$UC3_USERNAME/" > tmp && mv tmp si\ +tes.xml +fi +if [[ -z $BEAGLE_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/{beagle.USER}/$BEAGLE_USERNAME/" > tmp && mv \ +tmp sites.xml +fi +if [[ -z $MCS_USERNAME ]] +then + echo "Remote username not provided. Skipping sites configs" +else + cat sites.xml | sed "s/{mcs.USER}/$MCS_USERNAME/" > tmp && mv \ +tmp sites.xml +fi + +cat<<'EOF' > teragen_wrap.sh +#!/bin/bash + +# By default with ARG1:100 and SLICESIZE=10000, this script will generate +# 10^6 records. +ARG1=1 +[ ! -z $1 ] && ARG1=$1 + +FILE="input_$RANDOM.txt" +LOWERLIMIT=0 +UPPERLIMIT=1000000 # 10^9 +SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file +#SLICESIZE=1000 # 10^3 If padded to 100B would result + +shuf -i $LOWERLIMIT-$UPPERLIMIT -n $(($SLICESIZE*$ARG1)) | awk '{printf "%-99s\n", $0}' +exit 0 +EOF + +cat <<'EOF' > combiner.sh +#!/bin/bash + +FILES=$* +SUM=0 +COUNT=0 + +for file in $* +do + RES=($(awk '{ sum += $1 } END { print sum,NR }' $file)) + echo "${RES[0]} ${RES[1]}" + SUM=$(($SUM+${RES[0]})) + COUNT=$(($COUNT+${RES[1]})) +done +echo "SUM : $SUM" +echo "COUNT: $COUNT" +exit 0 +EOF Property changes on: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.setup.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.stdout =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.stdout (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.stdout 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,3 @@ +Swift 0.94.1 RC2 swift-r6895 cog-r3765 + +RunID: 20130820-2209-384c1ky1 Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.swift =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.swift (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.swift 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,30 @@ +type file; +type script; + +app (file out, file err) gen_data (script run, int recsize) +{ + bash @run recsize stdout=@out stderr=@err; +} + +app (file out, file err) comb_data (script comb, file array[]) +{ + bash @comb @array stdout=@out stderr=@err; +} + + +file tgen_out[] ; +file tgen_err[] ; + +script wrapper <"teragen_wrap.sh">; +int loop = @toInt(@arg("loops","10")); +int fsize = @toInt(@arg("recsize","1")); # This would make 10M records per file +string dir = @arg("dir", "./"); + +foreach item,i in [0:loop-1] { + (tgen_out[i], tgen_err[i]) = gen_data(wrapper, fsize); +} + +script combine <"combiner.sh">; +file final <"final_result">; +file errs <"err_file">; +(final, errs) = comb_data(combine, tgen_out); Added: branches/release-0.95/tests/stress/local_cluster/simple_MapRed.timeout =================================================================== --- branches/release-0.95/tests/stress/local_cluster/simple_MapRed.timeout (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/simple_MapRed.timeout 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1 @@ +900 \ No newline at end of file Added: branches/release-0.95/tests/stress/local_cluster/sites.template.xml =================================================================== --- branches/release-0.95/tests/stress/local_cluster/sites.template.xml (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/sites.template.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,66 @@ + + + + + 10.00 + 10000 + 1 + 3600 + 00:15:00 + 100 + 100 + 1000 + 1 + 1 + "group_friends.{uc3.USER}" + nonshared + . + + + + + 24 + 100 + 100 + pbs.aprun;pbs.mpp;depth=24 + 3700 + 01:00:00 + /lustre/beagle/{env.USER}/swiftwork + 1 + 2 + 1 + 1.00 + 10000 + trace + /tmp/{beagle.USER}/swiftwork + + + + + sandyb + 16 + 00:15:00 + 3600 + 100 + 100 + 4 + 1 + 1 + .64 + 10000 + /tmp/{mid.USER} + + + + + 8 + 100 + 100 + 3600 + 00:15:00 + 0.0799 + 10000 + /sandbox/{mcs.USER}/swiftwork + + + Added: branches/release-0.95/tests/stress/local_cluster/sites.xml =================================================================== --- branches/release-0.95/tests/stress/local_cluster/sites.xml (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/sites.xml 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,65 @@ + + + + + 10.00 + 10000 + 1 + 3600 + 00:15:00 + 100 + 100 + 4 + 1 + 1 + "group_friends.yadunand" + nonshared + . + + + + + 24 + 100 + 100 + + 3600 + 00:15:00 + /lustre/beagle/yadunandb/swiftwork + 4 + 1 + 1 + 1.00 + 10000 + /tmp/yadunandb/swiftwork + + + + + sandyb + 16 + 00:15:00 + 3600 + 100 + 100 + 4 + 1 + 1 + .64 + 10000 + /tmp/yadunand + + + + + 8 + 100 + 100 + 3600 + 00:15:00 + 0.0799 + 10000 + /sandbox/yadunand/swiftwork + + + Added: branches/release-0.95/tests/stress/local_cluster/swift.properties =================================================================== --- branches/release-0.95/tests/stress/local_cluster/swift.properties (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/swift.properties 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,8 @@ +use.provider.staging=true +use.wrapper.staging=false +wrapperlog.always.transfer=true +execution.retries=0 +lazy.errors=false +provider.staging.pin.swiftfiles=false +sitedir.keep=true +tcp.port.range=50000,51000 \ No newline at end of file Added: branches/release-0.95/tests/stress/local_cluster/teragen_wrap.sh =================================================================== --- branches/release-0.95/tests/stress/local_cluster/teragen_wrap.sh (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/teragen_wrap.sh 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,15 @@ +#!/bin/bash + +# By default with ARG1:100 and SLICESIZE=10000, this script will generate +# 10^6 records. +ARG1=1 +[ ! -z $1 ] && ARG1=$1 + +FILE="input_$RANDOM.txt" +LOWERLIMIT=0 +UPPERLIMIT=1000000 # 10^9 +SLICESIZE=10000 # 10^4 records padded to 100B would result in 1MB file +#SLICESIZE=1000 # 10^3 If padded to 100B would result + +shuf -i $LOWERLIMIT-$UPPERLIMIT -n $(($SLICESIZE*$ARG1)) | awk '{printf "%-99s\n", $0}' +exit 0 Property changes on: branches/release-0.95/tests/stress/local_cluster/teragen_wrap.sh ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/tests/stress/local_cluster/title.txt =================================================================== --- branches/release-0.95/tests/stress/local_cluster/title.txt (rev 0) +++ branches/release-0.95/tests/stress/local_cluster/title.txt 2014-06-14 21:07:38 UTC (rev 7923) @@ -0,0 +1,4 @@ +Simple MapReduce style job for Local cluster testing +| The first map stage generates a large number of random numbers +| The reduce stage aggregates the results and outputs the count +| and sum From swift at ci.uchicago.edu Wed Jun 18 22:30:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 18 Jun 2014 22:30:03 -0500 (CDT) Subject: [Swift-commit] cog r3919 Message-ID: <20140619033003.D9D128D000E9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3919 | hategan | 2014-06-18 22:29:00 -0500 (Wed, 18 Jun 2014) | 1 line properly redirect stdout and stderr from ssh when running multi-jobs ------------------------------------------------------------------------ Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java (revision 3918) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/lsf/LSFExecutor.java (working copy) @@ -208,7 +208,7 @@ } if (multiple) { - writeMultiJobPostamble(wr); + writeMultiJobPostamble(wr, stdout, stderr); } else { wr.write('\n'); wr.write("/bin/echo $? >" + exitcodefile + '\n'); Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java (revision 3918) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/pbs/PBSExecutor.java (working copy) @@ -310,7 +310,7 @@ wr.write(" < " + quote(spec.getStdInput())); } if (multiple) { - writeMultiJobPostamble(wr); + writeMultiJobPostamble(wr, stdout, stderr); } else { wr.write('\n'); Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (revision 3918) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (working copy) @@ -457,8 +457,13 @@ return sb.toString(); } - protected void writeMultiJobPostamble(Writer wr) throws IOException { - wr.write("; echo \\\\\\$? > $ECF.$INDEX \" \\\" &\n"); + protected void writeMultiJobPostamble(Writer wr, String stdout, String stderr) throws IOException { + wr.write("; echo \\\\\\$? > $ECF.$INDEX \" \\\" "); + wr.write("1>>"); + wr.write(quote(stdout)); + wr.write("2>>"); + wr.write(quote(stderr)); + wr.write(" &\n"); wr.write(" INDEX=$((INDEX + 1))\n"); wr.write("done\n"); wr.write("wait\n"); Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/SGEExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/SGEExecutor.java (revision 3918) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/sge/SGEExecutor.java (working copy) @@ -8,12 +8,10 @@ import java.io.BufferedReader; import java.io.CharArrayReader; -import java.io.File; import java.io.IOException; import java.io.Writer; import java.text.DecimalFormat; import java.text.NumberFormat; -import java.util.ArrayList; import java.util.Iterator; import java.util.List; import java.util.regex.Matcher; @@ -27,12 +25,10 @@ import org.globus.cog.abstraction.impl.scheduler.common.Job; import org.globus.cog.abstraction.impl.scheduler.common.ProcessException; import org.globus.cog.abstraction.impl.scheduler.common.ProcessListener; -import org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor; import org.globus.cog.abstraction.interfaces.FileLocation; import org.globus.cog.abstraction.interfaces.JobSpecification; import org.globus.cog.abstraction.interfaces.Task; import org.globus.gsi.gssapi.auth.AuthorizationException; -import org.ietf.jgss.GSSException; /** * Java CoG interface for Sun/Oracle Grid Engine @@ -446,7 +442,7 @@ } if (multiple) { - writeMultiJobPostamble(wr); + writeMultiJobPostamble(wr, stdout, stderr); } else { wr.write(" &\n"); wr.write("wait $!\n"); From swift at ci.uchicago.edu Thu Jun 19 13:00:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Thu, 19 Jun 2014 13:00:04 -0500 (CDT) Subject: [Swift-commit] cog r3921 Message-ID: <20140619180005.27D2A8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3921 | hategan | 2014-06-19 12:57:12 -0500 (Thu, 19 Jun 2014) | 1 line fixed space before stderr redirect ------------------------------------------------------------------------ Index: modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java =================================================================== --- modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (revision 3920) +++ modules/provider-localscheduler/src/org/globus/cog/abstraction/impl/scheduler/common/AbstractExecutor.java (working copy) @@ -461,7 +461,7 @@ wr.write("; echo \\\\\\$? > $ECF.$INDEX \" \\\" "); wr.write("1>>"); wr.write(quote(stdout)); - wr.write("2>>"); + wr.write(" 2>>"); wr.write(quote(stderr)); wr.write(" &\n"); wr.write(" INDEX=$((INDEX + 1))\n"); From swift at ci.uchicago.edu Fri Jun 20 17:05:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 20 Jun 2014 17:05:03 -0500 (CDT) Subject: [Swift-commit] cog r3935 Message-ID: <20140620220503.9630E8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3935 | timgarmstrong | 2014-06-20 17:03:10 -0500 (Fri, 20 Jun 2014) | 1 line Set redirects ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3934) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -236,6 +236,30 @@ delete job; } +coaster_rc +coaster_job_set_redirects(coaster_job *job, const char *stdin_loc, + const char *stdout_loc, const char *stderr_loc) + COASTERS_THROWS_NOTHING { + if (job == NULL) { + return COASTER_ERROR_INVALID; + } + + // Accessor methods shouldn't throw exceptions + std::string *stdin_str = (stdin_loc == NULL) ? + NULL : new string(stdin_loc); + job->job.setStdinLocation(*stdin_str); + + std::string *stdout_str = (stdout_loc == NULL) ? + NULL : new string(stdout_loc); + job->job.setStdinLocation(*stdout_str); + + std::string *stderr_str = (stderr_loc == NULL) ? + NULL : new string(stderr_loc); + job->job.setStdinLocation(*stderr_str); + + return COASTER_SUCCESS; +} + const char * coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING { // Shouldn't throw anything from accessor method Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3934) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -41,6 +41,7 @@ */ typedef enum { COASTER_SUCCESS, + COASTER_ERROR_INVALID, COASTER_ERROR_OOM, COASTER_ERROR_NETWORK, COASTER_ERROR_UNKNOWN, @@ -130,7 +131,6 @@ * Create a new coasters job for later submission. * Some standard arguments can be specified now, or left as NULL to be * initialized later. - * TODO: functions to set other job properties * * executable: must be provided, name of executable * argc/argv: command line arguments @@ -148,7 +148,18 @@ coaster_rc coaster_job_free(coaster_job *job) COASTERS_THROWS_NOTHING; +/* + * Set input and output streams redirections. + * All can be NULL to use default. + */ +coaster_rc +coaster_job_set_redirects(coaster_job *job, const char *stdin_loc, + const char *stdout_loc, const char *stderr_loc) + COASTERS_THROWS_NOTHING; +// TODO: functions for setting directory, env vars, attributes, +// stageins, stageouts, cleanups + /* * Get local job ID string. * Return value points to job state: will be invalid if job is freed. @@ -167,6 +178,7 @@ * Get name of return code. Returns NULL if invalid code. */ const char *coaster_rc_string(coaster_rc code); + #ifdef __cplusplus } // extern "C" #endif From swift at ci.uchicago.edu Fri Jun 20 18:20:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 20 Jun 2014 18:20:03 -0500 (CDT) Subject: [Swift-commit] cog r3936 Message-ID: <20140620232003.D70D88D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3936 | timgarmstrong | 2014-06-20 18:17:27 -0500 (Fri, 20 Jun 2014) | 1 line Add a dose of pedantry to the coaster C client interfaces based on my attempts to understand memory lifetime/ownership. Document in a few cases and add const qualifiers where appropriate in other cases. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterChannel.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.h (revision 3935) +++ modules/provider-coaster-c-client/src/CoasterChannel.h (working copy) @@ -103,7 +103,7 @@ void send(int tag, Buffer* buf, int flags, ChannelCallback* cb); CoasterClient* getClient(); - string& getURL(); + const string& getURL() const; void checkHeartbeat(); Index: modules/provider-coaster-c-client/src/CoasterLoop.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterLoop.h (revision 3935) +++ modules/provider-coaster-c-client/src/CoasterLoop.h (working copy) @@ -51,7 +51,12 @@ virtual ~CoasterLoop(); void start(); void stop(); - + + /* + * Add a channel for the loop to monitor. + * Ownership of the channel is retained by the caller. + * Must be removed later by a call to removeChannel(). + */ void addChannel(CoasterChannel* channel); void removeChannel(CoasterChannel* channel); void addSockets(); Index: modules/provider-coaster-c-client/src/CoasterClient.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.cpp (revision 3935) +++ modules/provider-coaster-c-client/src/CoasterClient.cpp (working copy) @@ -167,7 +167,7 @@ delete cmd; } -void CoasterClient::updateJobStatus(string& remoteJobId, JobStatus* status) { Lock::Scoped l(lock); +void CoasterClient::updateJobStatus(const string& remoteJobId, JobStatus* status) { Lock::Scoped l(lock); if (remoteJobIdMapping.count(remoteJobId) == 0) { LogWarn << "Received job status notification for unknown job (" << remoteJobId << "): " << status << endl; } @@ -259,12 +259,12 @@ } } -void CoasterClient::waitForJob(Job& job) { Lock::Scoped l(lock); +void CoasterClient::waitForJob(const Job& job) { Lock::Scoped l(lock); while (jobs.count(&(job.getIdentity())) != 0) { cv.wait(lock); } } -string& CoasterClient::getURL() { +const string& CoasterClient::getURL() { return URL; } Index: modules/provider-coaster-c-client/src/JobStatus.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobStatus.cpp (revision 3935) +++ modules/provider-coaster-c-client/src/JobStatus.cpp (working copy) @@ -44,7 +44,7 @@ init(UNSUBMITTED, time(NULL), NULL, NULL); } -JobStatusCode JobStatus::getStatusCode() { +JobStatusCode JobStatus::getStatusCode() const { return statusCode; } @@ -52,7 +52,7 @@ return stime; } -string* JobStatus::getMessage() { +const string* JobStatus::getMessage() const { return message; } Index: modules/provider-coaster-c-client/src/Job.h =================================================================== --- modules/provider-coaster-c-client/src/Job.h (revision 3935) +++ modules/provider-coaster-c-client/src/Job.h (working copy) @@ -27,14 +27,16 @@ * TODO: document expectations about lifetime of strings. * It seems very easy for client code to accidentally * pass in a pointer to a stack-allocated string or a - * string with a shorter lifetime than the job. + * string with a shorter lifetime than the job. Would + * it work to just store them by value and have zero-length + * be equivalent to NULL. Are zero-length strings meaningful? */ vector* arguments; string* directory; string* stdinLocation; string* stdoutLocation; string* stderrLocation; - string jobManager; + string jobManager; map* env; map* attributes; @@ -49,41 +51,41 @@ Job(const string &executable); virtual ~Job(); - string& getExecutable(); + const string& getExecutable() const; /* Get the job identity. The identity is a unique string that is assigned to the job object upon construction and does not change over it's lifetime. */ - string& getIdentity(); + const string& getIdentity() const; vector* getArguments(); void addArgument(string& arg); void addArgument(const char* arg); - string* getDirectory(); + const string* getDirectory() const; void setDirectory(string& directory); - string* getStdinLocation(); + const string* getStdinLocation() const; void setStdinLocation(string& loc); - string* getStdoutLocation(); + const string* getStdoutLocation() const; void setStdoutLocation(string& loc); - string* getStderrLocation(); + const string* getStderrLocation() const; void setStderrLocation(string& loc); - string getJobManager(); + const string &getJobManager() const; void setJobManager(string jm); void setJobManager(const char *jm); map* getEnv(); - string* getEnv(string name); + const string* getEnv(string name) const; void setEnv(string name, string value); map* getAttributes(); - string* getAttribute(string name); + const string* getAttribute(string name); void setAttribute(string name, string value); vector* getStageIns(); @@ -95,10 +97,7 @@ vector* getCleanups(); void addCleanup(string cleanup); - string* getStderr(); - string* getStdout(); - - JobStatus* getStatus(); + const JobStatus* getStatus() const; void setStatus(JobStatus* status); }; Index: modules/provider-coaster-c-client/src/CoasterChannel.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.cpp (revision 3935) +++ modules/provider-coaster-c-client/src/CoasterChannel.cpp (working copy) @@ -326,8 +326,8 @@ delete cmd; } -string& CoasterChannel::getURL() { - return getClient()->getURL(); +const string& CoasterChannel::getURL() const { + return client->getURL(); } /* Index: modules/provider-coaster-c-client/src/CoasterClient.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.h (revision 3935) +++ modules/provider-coaster-c-client/src/CoasterClient.h (working copy) @@ -63,15 +63,31 @@ void stop(); void setOptions(Settings& settings); + + /* + * Submit a job. The job should have been filled in with + * all properties. The ownership of the job object stays + * with the caller, but this client will retain a reference + * to the job until done jobs are purged. + */ void submit(Job& job); - void waitForJob(Job& job); + /* + * Wait for job to be done. Upon completion no actions + * are taken: job must be purged from client explicitly. + */ + void waitForJob(const Job& job); + + /* + * Return a list of done jobs and remove references from this + * client. + */ list* getAndPurgeDoneJobs(); void waitForJobs(); - void updateJobStatus(string& jobId, JobStatus* status); + void updateJobStatus(const string &jobId, JobStatus* status); - string& getURL(); + const string& getURL(); void errorReceived(Command* cmd, string* message, RemoteCoasterException* details); void replyReceived(Command* cmd); Index: modules/provider-coaster-c-client/src/Job.cpp =================================================================== --- modules/provider-coaster-c-client/src/Job.cpp (revision 3935) +++ modules/provider-coaster-c-client/src/Job.cpp (working copy) @@ -32,7 +32,7 @@ status = NULL; } -string& Job::getIdentity() { +const string& Job::getIdentity() const { return identity; } @@ -47,7 +47,7 @@ arguments->push_back(new string(arg)); } -string& Job::getExecutable() { +const string& Job::getExecutable() const { return executable; } @@ -55,7 +55,7 @@ addArgument(*(new string(arg))); } -string* Job::getDirectory() { +const string* Job::getDirectory() const { return directory; } @@ -63,7 +63,7 @@ directory = &pdirectory; } -string* Job::getStdinLocation() { +const string* Job::getStdinLocation() const { return stdinLocation; } @@ -71,7 +71,7 @@ stdinLocation = &loc; } -string* Job::getStdoutLocation() { +const string* Job::getStdoutLocation() const { return stdoutLocation; } @@ -79,7 +79,7 @@ stdoutLocation = &loc; } -string* Job::getStderrLocation() { +const string* Job::getStderrLocation() const { return stderrLocation; } @@ -87,7 +87,7 @@ stderrLocation = &loc; } -string Job::getJobManager() { +const string& Job::getJobManager() const { return jobManager; } @@ -104,7 +104,7 @@ return env; } -string* Job::getEnv(string name) { +const string* Job::getEnv(string name) const { if (env == NULL) { return NULL; } @@ -131,7 +131,7 @@ return attributes; } -string* Job::getAttribute(string name) { +const string* Job::getAttribute(string name) { if (attributes == NULL) { return NULL; } @@ -187,15 +187,7 @@ cleanups->push_back(cleanup); } -string* Job::getStderr() { - return stderr; -} - -string* Job::getStdout() { - return stdout; -} - -JobStatus* Job::getStatus() { +const JobStatus* Job::getStatus() const { return status; } Index: modules/provider-coaster-c-client/src/JobStatus.h =================================================================== --- modules/provider-coaster-c-client/src/JobStatus.h (revision 3935) +++ modules/provider-coaster-c-client/src/JobStatus.h (working copy) @@ -47,9 +47,9 @@ JobStatus(JobStatusCode statusCode); JobStatus(); virtual ~JobStatus(); - JobStatusCode getStatusCode(); + JobStatusCode getStatusCode() const; time_t getTime(); - string* getMessage(); + const string* getMessage() const; RemoteCoasterException* getException(); const JobStatus* getPreviousStatus(); void setPreviousStatus(JobStatus* prev); Index: modules/provider-coaster-c-client/src/JobSubmitCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (revision 3935) +++ modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (working copy) @@ -5,8 +5,8 @@ using namespace std; -void add(string& ss, const char* key, string* value); -void add(string& ss, const char* key, string& value); +void add(string& ss, const char* key, const string* value); +void add(string& ss, const char* key, const string& value); void add(string& ss, const char* key, const char* value); void add(string& ss, const char* key, const char* value, int n); @@ -75,13 +75,13 @@ addOutData(Buffer::wrap(ss)); } -void add(string& ss, const char* key, string* value) { +void add(string& ss, const char* key, const string* value) { if (value != NULL) { add(ss, key, value->data(), value->length()); } } -void add(string& ss, const char* key, string& value) { +void add(string& ss, const char* key, const string& value) { add(ss, key, value.data(), value.length()); } From swift at ci.uchicago.edu Fri Jun 20 18:50:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Fri, 20 Jun 2014 18:50:03 -0500 (CDT) Subject: [Swift-commit] cog r3938 Message-ID: <20140620235003.BDA378D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3938 | timgarmstrong | 2014-06-20 18:45:58 -0500 (Fri, 20 Jun 2014) | 1 line Implement a few more functions to modify job ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3937) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -62,6 +62,11 @@ static coaster_rc coaster_error_rc(const CoasterError &err); static coaster_rc exception_rc(const std::exception &ex); +// Check for error +// TODO: store error message somewhere instead of printing? +#define COASTER_CONDITION(cond, err_rc, err_msg) { \ + if (!(cond)) { fprintf(stderr, (err_msg)); return (err_rc); }} + coaster_rc coaster_client_start(const char *serviceURL, coaster_client **client) COASTERS_THROWS_NOTHING { @@ -244,22 +249,101 @@ return COASTER_ERROR_INVALID; } - // Accessor methods shouldn't throw exceptions - std::string *stdin_str = (stdin_loc == NULL) ? - NULL : new string(stdin_loc); - job->job.setStdinLocation(*stdin_str); + try { + std::string *stdin_str = (stdin_loc == NULL) ? + NULL : new string(stdin_loc); + job->job.setStdinLocation(*stdin_str); + + std::string *stdout_str = (stdout_loc == NULL) ? + NULL : new string(stdout_loc); + job->job.setStdinLocation(*stdout_str); + + std::string *stderr_str = (stderr_loc == NULL) ? + NULL : new string(stderr_loc); + job->job.setStdinLocation(*stderr_str); + + return COASTER_SUCCESS; + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } +} + +coaster_rc +coaster_job_set_directory(coaster_job *job, const char *dir) + COASTERS_THROWS_NOTHING { + if (job == NULL) { + return COASTER_ERROR_INVALID; + } - std::string *stdout_str = (stdout_loc == NULL) ? - NULL : new string(stdout_loc); - job->job.setStdinLocation(*stdout_str); + try { + std::string *dir_str = (dir == NULL) ? + NULL : new string(dir); + job->job.setDirectory(*dir_str); + + return COASTER_SUCCESS; + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } +} + +coaster_rc +coaster_job_set_envs(coaster_job *job, int nvars, const char **names, + const char **values) COASTERS_THROWS_NOTHING { + if (job == NULL) { + return COASTER_ERROR_INVALID; + } - std::string *stderr_str = (stderr_loc == NULL) ? - NULL : new string(stderr_loc); - job->job.setStdinLocation(*stderr_str); + try { + for (int i = 0; i < nvars; i++) + { + const char *name = names[i]; + const char *value = values[i]; + COASTER_CONDITION(name != NULL && value != NULL, + COASTER_ERROR_INVALID, "Env var name or value was NULL"); + job->job.setEnv(name, value); + } - return COASTER_SUCCESS; + return COASTER_SUCCESS; + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } } +/* + * Add attributes for the job. Will overwrite any previous atrributes + * if names match. + */ +coaster_rc +coaster_job_set_attrs(coaster_job *job, int nattrs, const char **names, + const char **values) COASTERS_THROWS_NOTHING { + if (job == NULL) { + return COASTER_ERROR_INVALID; + } + + try { + for (int i = 0; i < nattrs; i++) + { + const char *name = names[i]; + const char *value = values[i]; + COASTER_CONDITION(name != NULL && value != NULL, + COASTER_ERROR_INVALID, "Attribute name or value was NULL"); + job->job.setAttribute(name, value); + } + + return COASTER_SUCCESS; + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } +} + const char * coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING { // Shouldn't throw anything from accessor method @@ -308,5 +392,6 @@ static coaster_rc exception_rc(const std::exception &ex) { // TODO: store error info? + // TODO: handle specific types, e.g. bad_alloc return COASTER_ERROR_UNKNOWN; } Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3937) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -157,10 +157,34 @@ const char *stdout_loc, const char *stderr_loc) COASTERS_THROWS_NOTHING; -// TODO: functions for setting directory, env vars, attributes, -// stageins, stageouts, cleanups +/* + * Set job directory. + */ +coaster_rc +coaster_job_set_directory(coaster_job *job, const char *dir) + COASTERS_THROWS_NOTHING; /* + * Add environment variables for the job. Will overwrite any + * previous values if names match. + * name and value strings should not be NULL. + */ +coaster_rc +coaster_job_set_envs(coaster_job *job, int nvars, const char **names, + const char **values) COASTERS_THROWS_NOTHING; + +/* + * Add attributes for the job. Will overwrite any previous atrributes + * if names match. + * name and value strings should not be NULL. + */ +coaster_rc +coaster_job_set_attrs(coaster_job *job, int nattrs, const char **names, + const char **values) COASTERS_THROWS_NOTHING; + +// TODO: functions for setting stageins, stageouts, cleanups + +/* * Get local job ID string. * Return value points to job state: will be invalid if job is freed. */ From davidk at ci.uchicago.edu Fri Jun 20 19:42:00 2014 From: davidk at ci.uchicago.edu (davidk at ci.uchicago.edu) Date: Fri, 20 Jun 2014 19:42:00 -0500 (CDT) Subject: [Swift-commit] r7924 - in branches/release-0.95: . bin Message-ID: <20140621004200.CD3499CC72@svn.ci.uchicago.edu> Author: davidk Date: 2014-06-20 19:42:00 -0500 (Fri, 20 Jun 2014) New Revision: 7924 Added: branches/release-0.95/bin/moab-qdel branches/release-0.95/bin/moab-qstat branches/release-0.95/bin/moab-qsub Modified: branches/release-0.95/build.xml Log: Moab wrappers Added: branches/release-0.95/bin/moab-qdel =================================================================== --- branches/release-0.95/bin/moab-qdel (rev 0) +++ branches/release-0.95/bin/moab-qdel 2014-06-21 00:42:00 UTC (rev 7924) @@ -0,0 +1,3 @@ +#!/bin/bash + +canceljob "$@" Property changes on: branches/release-0.95/bin/moab-qdel ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/bin/moab-qstat =================================================================== --- branches/release-0.95/bin/moab-qstat (rev 0) +++ branches/release-0.95/bin/moab-qstat 2014-06-21 00:42:00 UTC (rev 7924) @@ -0,0 +1,32 @@ +#!/usr/bin/python + +import sys +import subprocess +import getpass + +jobs = sys.argv[2:len(sys.argv)] + +username = getpass.getuser() +p = subprocess.Popen(['showq', '-w', 'user='+username], stdout=subprocess.PIPE, stderr=subprocess.PIPE) +out, err = p.communicate() +out_list = out.rstrip().split('\n') + +for o in out_list: + try: + words = o.split() + jobid = words[0].rstrip() + jobstatus = words[2].rstrip() + + if jobid in jobs: + if jobstatus in ["Idle", "Deferred", "BatchHold", "SystemHold", "UserHold", "Deferred", "Migrated", "Staging", "Starting"]: + jobstatus = "Q" + elif jobstatus in ["Running"]: + jobstatus = "R" + elif jobstatus == ["Completed", "Vacated"]: + jobstatus = "C" + else: + continue + print "Job Id: " + jobid + print "\tjob_state = " + jobstatus + "\n" + except: + continue Property changes on: branches/release-0.95/bin/moab-qstat ___________________________________________________________________ Added: svn:executable + * Added: branches/release-0.95/bin/moab-qsub =================================================================== --- branches/release-0.95/bin/moab-qsub (rev 0) +++ branches/release-0.95/bin/moab-qsub 2014-06-21 00:42:00 UTC (rev 7924) @@ -0,0 +1,3 @@ +#!/bin/bash + +msub "$@" Property changes on: branches/release-0.95/bin/moab-qsub ___________________________________________________________________ Added: svn:executable + * Modified: branches/release-0.95/build.xml =================================================================== --- branches/release-0.95/build.xml 2014-06-14 21:07:38 UTC (rev 7923) +++ branches/release-0.95/build.xml 2014-06-21 00:42:00 UTC (rev 7924) @@ -87,6 +87,9 @@ + + + From swift at ci.uchicago.edu Mon Jun 23 09:10:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 09:10:03 -0500 (CDT) Subject: [Swift-commit] cog r3947 Message-ID: <20140623141003.91A998D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3947 | timgarmstrong | 2014-06-23 09:06:14 -0500 (Mon, 23 Jun 2014) | 1 line Use typedefed type ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3946) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -400,7 +400,7 @@ } } -int64_t +job_id_t coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING { // Shouldn't throw anything from accessor method return job->job.getIdentity(); From swift at ci.uchicago.edu Mon Jun 23 09:20:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 09:20:04 -0500 (CDT) Subject: [Swift-commit] cog r3948 Message-ID: <20140623142004.C6C938D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3948 | timgarmstrong | 2014-06-23 09:17:48 -0500 (Mon, 23 Jun 2014) | 1 line Little documentation and a note about object lifetime for settings. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h =================================================================== --- modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h (revision 3947) +++ modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h (working copy) @@ -14,6 +14,7 @@ Settings* settings; public: static string NAME; + // TODO: how long does this hold a reference to settings? ServiceConfigurationCommand(Settings& s); virtual void send(CoasterChannel* channel, CommandCallback* cb); private: Index: modules/provider-coaster-c-client/src/CoasterClient.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.h (revision 3947) +++ modules/provider-coaster-c-client/src/CoasterClient.h (working copy) @@ -60,7 +60,8 @@ virtual ~CoasterClient(); void start(); void stop(); - + + // TODO: how long does this hold a reference to settings? void setOptions(Settings& settings); /* Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3947) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -20,7 +20,9 @@ * Created: Jun 18, 2014 * Author: Tim Armstrong * - * Pure C interface for Coasters + * Pure C interface for Coasters. This aims to expose a subset of the + * C++ API's functionality that is sufficient to support all common use + * cases for submitting jobs through Coasters. */ #ifndef COASTERS_H_ @@ -56,7 +58,6 @@ } coaster_rc; // Set appropriate macro to specify that we shouldn't throw exceptions -// Only used in this header: undefine later #ifdef __cplusplus #define COASTERS_THROWS_NOTHING throw() #else @@ -75,7 +76,10 @@ coaster_client **client) COASTERS_THROWS_NOTHING; /* - * Stop coasters client and free memory. + * Stop coasters client and free all memory. + * + * After calling this the client is invalid and should not be used as + * an argument to any more Coaster C API function calls. */ coaster_rc coaster_client_stop(coaster_client *client) COASTERS_THROWS_NOTHING; @@ -90,6 +94,13 @@ COASTERS_THROWS_NOTHING; /* + * Free memory associated with coasters settings + */ +void +coaster_settings_free(coaster_settings *settings) + COASTERS_THROWS_NOTHING; + +/* * Parse settings from string. * * str[_len]: String with key/value settings and length of string. @@ -130,12 +141,10 @@ const char ***keys, size_t **key_lens, int *count) COASTERS_THROWS_NOTHING; -void -coaster_settings_free(coaster_settings *settings) - COASTERS_THROWS_NOTHING; - /* * Apply settings to started coasters client. + * TODO: currently it isn't safe to free settings until client is shut + * down */ coaster_rc coaster_apply_settings(coaster_client *client, @@ -165,7 +174,7 @@ coaster_job_free(coaster_job *job) COASTERS_THROWS_NOTHING; /* - * Set input and output streams redirections. + * Set input and output stream redirections. * If set to NULL, don't modify. */ coaster_rc @@ -214,13 +223,14 @@ // TODO: functions for setting stageins, stageouts /* - * Get local job ID. + * Get local job ID. The job ID is a locally unique identifier for + * a coasters job that is assigned when the job is created. */ job_id_t coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING; /* - * Get status of job. + * Get status of a submitted job. * Return COASTER_ERROR_INVALID if job is invalid or has no status. */ coaster_rc @@ -239,7 +249,7 @@ COASTERS_THROWS_NOTHING; /* - * Submit a coasters job. + * Submit a coasters job through a coasters client. * A job can only be submitted once! */ coaster_rc From swift at ci.uchicago.edu Mon Jun 23 10:50:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 10:50:03 -0500 (CDT) Subject: [Swift-commit] cog r3949 Message-ID: <20140623155003.B9CAB8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3949 | timgarmstrong | 2014-06-23 10:48:26 -0500 (Mon, 23 Jun 2014) | 1 line Add C API functionality to wait for jobs. Minor additions to C++ API to wait for single completion of any job and to get bounded number of completed jobs ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterClient.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.cpp (revision 3948) +++ modules/provider-coaster-c-client/src/CoasterClient.cpp (working copy) @@ -276,12 +276,28 @@ } } +int CoasterClient::getAndPurgeDoneJobs(int size, Job** jobs) + { Lock::Scoped l(lock); + int count = 0; + while (count < size && !doneJobs.empty()) { + jobs[count++] = doneJobs.front(); + doneJobs.pop_front(); + } + return count; +} + void CoasterClient::waitForJobs() { Lock::Scoped l(lock); while (jobs.size() != 0) { cv.wait(lock); } } +void CoasterClient::waitForAnyJob() { Lock::Scoped l(lock); + while (doneJobs.size() == 0) { + cv.wait(lock); + } +} + void CoasterClient::waitForJob(const Job& job) { Lock::Scoped l(lock); while (jobs.count(job.getIdentity()) != 0) { cv.wait(lock); Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3948) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -461,6 +461,51 @@ } } +coaster_rc +coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, + job_id_t *jobs, int *njobs) + COASTERS_THROWS_NOTHING { + if (client == NULL) { + return COASTER_ERROR_INVALID; + } + + try { + if (wait) { + client->client.waitForAnyJob(); + } + + int count = 0; + + // Need to use temporary storage for job pointers + const int job_buf_size = 32; + Job *job_buf[job_buf_size]; + + while (count < maxjobs) { + int maxleft = maxjobs - count; + int maxbatch = (maxleft < job_buf_size) ? maxleft : job_buf_size; + + int n = client->client.getAndPurgeDoneJobs(maxbatch, job_buf); + + for (int i = 0; i < n; i++) { + jobs[count++] = job_buf[i]->getIdentity(); + } + + if (n < maxbatch) { + // Got last job + break; + } + } + + *njobs = count; + return COASTER_SUCCESS; + + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } +} + const char *coaster_rc_string(coaster_rc code) { switch (code) Index: modules/provider-coaster-c-client/src/CoasterClient.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.h (revision 3948) +++ modules/provider-coaster-c-client/src/CoasterClient.h (working copy) @@ -60,8 +60,8 @@ virtual ~CoasterClient(); void start(); void stop(); - - // TODO: how long does this hold a reference to settings? + + // TODO: how long does this hold a reference to settings? void setOptions(Settings& settings); /* @@ -80,10 +80,22 @@ /* * Return a list of done jobs and remove references from this - * client. + * client. This returns jobs in FIFO order of completion */ list* getAndPurgeDoneJobs(); + + /* + * Give back up to size done jobs and remove references. + * jobs: array with space for at least size jobs + * returns number of completed jobs added to array + */ + int getAndPurgeDoneJobs(int size, Job** jobs); + void waitForJobs(); + /* + * Wait until there is at least one done job + */ + void waitForAnyJob(); void updateJobStatus(const string& remoteJobId, JobStatus* status); void updateJobStatus(job_id_t jobId, JobStatus* status); Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3948) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -256,6 +256,21 @@ coaster_submit(coaster_client *client, coaster_job *job) COASTERS_THROWS_NOTHING; +/* + * Check for completion of jobs. + * + * NOTE: we only return job ids, client is responsible for reconciling + * these with the job objects. + * + * wait: if true, don't return until at least one job completes + * maxjobs: maximum number of jobs to return + * jobs: output array large enough to hold maxjobs + * njobs: output for number of jobs returned in jobs + */ +coaster_rc +coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, + job_id_t *jobs, int *njobs) + COASTERS_THROWS_NOTHING; /* * Get name of return code. Returns NULL if invalid code. From swift at ci.uchicago.edu Mon Jun 23 11:20:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 11:20:03 -0500 (CDT) Subject: [Swift-commit] cog r3950 Message-ID: <20140623162003.D5EB18D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3950 | timgarmstrong | 2014-06-23 11:19:35 -0500 (Mon, 23 Jun 2014) | 1 line Return pointers to jobs instead of ids ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3949) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -57,12 +57,6 @@ coaster_settings() : settings() {}; }; -struct coaster_job { - Job job; - - coaster_job(const string &executable) : job(executable) {}; -}; - static coaster_rc coaster_error_rc(const CoasterError &err); static coaster_rc exception_rc(const std::exception &ex); @@ -233,16 +227,16 @@ coaster_job **job) COASTERS_THROWS_NOTHING { try { assert(executable != NULL); - coaster_job *j = new coaster_job(string(executable, executable_len)); + Job *j = new Job(string(executable, executable_len)); for (int i = 0; i < argc; i++) { assert(argv[i] != NULL); - j->job.addArgument(argv[i], arg_lens[i]); + j->addArgument(argv[i], arg_lens[i]); } if (job_manager != NULL) { - j->job.setJobManager(job_manager, job_manager_len); + j->setJobManager(job_manager, job_manager_len); } *job = j; @@ -273,15 +267,15 @@ try { // job expects to get ownership of references, so use new if (stdin_loc != NULL) { - job->job.setStdinLocation(*new string(stdin_loc, stdin_loc_len)); + job->setStdinLocation(*new string(stdin_loc, stdin_loc_len)); } if (stdout_loc != NULL) { - job->job.setStdoutLocation(*new string(stdout_loc, stdout_loc_len)); + job->setStdoutLocation(*new string(stdout_loc, stdout_loc_len)); } if (stderr_loc != NULL) { - job->job.setStderrLocation(*new string(stderr_loc, stderr_loc_len)); + job->setStderrLocation(*new string(stderr_loc, stderr_loc_len)); } return COASTER_SUCCESS; @@ -301,7 +295,7 @@ try { if (dir != NULL) { - job->job.setDirectory(*new string(dir, dir_len)); + job->setDirectory(*new string(dir, dir_len)); } return COASTER_SUCCESS; @@ -329,7 +323,7 @@ size_t value_len = value_lens[i]; COASTER_CONDITION(name != NULL && value != NULL, COASTER_ERROR_INVALID, "Env var name or value was NULL"); - job->job.setEnv(name, name_len, value, value_len); + job->setEnv(name, name_len, value, value_len); } return COASTER_SUCCESS; @@ -362,7 +356,7 @@ size_t value_len = value_lens[i]; COASTER_CONDITION(name != NULL && value != NULL, COASTER_ERROR_INVALID, "Attribute name or value was NULL"); - job->job.setAttribute(name, name_len, value, value_len); + job->setAttribute(name, name_len, value, value_len); } return COASTER_SUCCESS; @@ -389,7 +383,7 @@ size_t cleanup_len = cleanup_lens[i]; COASTER_CONDITION(cleanup != NULL, COASTER_ERROR_INVALID, "Cleanup was NULL"); - job->job.addCleanup(cleanup, cleanup_len); + job->addCleanup(cleanup, cleanup_len); } return COASTER_SUCCESS; @@ -403,14 +397,14 @@ job_id_t coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING { // Shouldn't throw anything from accessor method - return job->job.getIdentity(); + return job->getIdentity(); } coaster_rc coaster_job_status_code(coaster_job *job, coaster_job_status *code) COASTERS_THROWS_NOTHING { const JobStatus *status; - if (job == NULL || (status = job->job.getStatus()) == NULL) { + if (job == NULL || (status = job->getStatus()) == NULL) { return COASTER_ERROR_INVALID; } @@ -427,8 +421,8 @@ return COASTER_ERROR_INVALID; } - const string *out = job->job.getStdout(); - const string *err = job->job.getStderr(); + const string *out = job->getStdout(); + const string *err = job->getStderr(); if (out != NULL) { *stderr_s = out->c_str(); @@ -453,7 +447,7 @@ coaster_submit(coaster_client *client, coaster_job *job) COASTERS_THROWS_NOTHING { try { - client->client.submit(job->job); + client->client.submit(*job); } catch (const CoasterError& err) { return coaster_error_rc(err); } catch (const std::exception& ex) { @@ -463,7 +457,7 @@ coaster_rc coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, - job_id_t *jobs, int *njobs) + coaster_job **jobs, int *njobs) COASTERS_THROWS_NOTHING { if (client == NULL) { return COASTER_ERROR_INVALID; @@ -474,29 +468,9 @@ client->client.waitForAnyJob(); } - int count = 0; + int n = client->client.getAndPurgeDoneJobs(maxjobs, jobs); - // Need to use temporary storage for job pointers - const int job_buf_size = 32; - Job *job_buf[job_buf_size]; - - while (count < maxjobs) { - int maxleft = maxjobs - count; - int maxbatch = (maxleft < job_buf_size) ? maxleft : job_buf_size; - - int n = client->client.getAndPurgeDoneJobs(maxbatch, job_buf); - - for (int i = 0; i < n; i++) { - jobs[count++] = job_buf[i]->getIdentity(); - } - - if (n < maxbatch) { - // Got last job - break; - } - } - - *njobs = count; + *njobs = n; return COASTER_SUCCESS; } catch (const CoasterError& err) { Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3949) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -43,8 +43,15 @@ // Opaque pointer types typedef struct coaster_client coaster_client; typedef struct coaster_settings coaster_settings; -typedef struct coaster_job coaster_job; +#ifdef __cplusplus +// Treat as direct pointer to class for C++ +typedef class Job coaster_job; +#else +// Treat as opaque pointer to unimplemented struct for C +typedef struct coaster_job_opaque_ coaster_job; +#endif + /* * Return codes for coaster errors * TODO: way to pass back error messages? @@ -259,9 +266,6 @@ /* * Check for completion of jobs. * - * NOTE: we only return job ids, client is responsible for reconciling - * these with the job objects. - * * wait: if true, don't return until at least one job completes * maxjobs: maximum number of jobs to return * jobs: output array large enough to hold maxjobs @@ -269,7 +273,7 @@ */ coaster_rc coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, - job_id_t *jobs, int *njobs) + coaster_job **jobs, int *njobs) COASTERS_THROWS_NOTHING; /* From swift at ci.uchicago.edu Mon Jun 23 11:30:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 11:30:03 -0500 (CDT) Subject: [Swift-commit] cog r3951 Message-ID: <20140623163003.96AED8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3951 | timgarmstrong | 2014-06-23 11:29:22 -0500 (Mon, 23 Jun 2014) | 1 line Ditch wrapper struct for settings ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3950) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -51,12 +51,6 @@ }; }; -struct coaster_settings { - Settings settings; - - coaster_settings() : settings() {}; -}; - static coaster_rc coaster_error_rc(const CoasterError &err); static coaster_rc exception_rc(const std::exception &ex); @@ -103,7 +97,7 @@ coaster_rc coaster_settings_create(coaster_settings **settings) COASTERS_THROWS_NOTHING { try { - *settings = new coaster_settings(); + *settings = new Settings(); if (!(*settings)) { return COASTER_ERROR_OOM; } @@ -129,7 +123,7 @@ const char *key, size_t key_len, const char *value, size_t value_len) COASTERS_THROWS_NOTHING { try { - settings->settings.set(key, key_len, value, value_len); + settings->set(key, key_len, value, value_len); return COASTER_SUCCESS; } catch (const CoasterError& err) { return coaster_error_rc(err); @@ -143,7 +137,7 @@ const char *key, size_t key_len, const char **value, size_t *value_len) COASTERS_THROWS_NOTHING { try { - std::map &map = settings->settings.getSettings(); + std::map &map = settings->getSettings(); std::string &str_value = map[string(key, key_len)]; *value = str_value.c_str(); *value_len = str_value.length(); @@ -160,7 +154,7 @@ const char ***keys, size_t **key_lens, int *count) COASTERS_THROWS_NOTHING { try { - std::map &map = settings->settings.getSettings(); + std::map &map = settings->getSettings(); *count = map.size(); // Use malloc so C client code can free @@ -210,8 +204,12 @@ coaster_apply_settings(coaster_client *client, coaster_settings *settings) COASTERS_THROWS_NOTHING { + if (settings == NULL || client == NULL) { + return COASTER_ERROR_INVALID; + } + try { - client->client.setOptions(settings->settings); + client->client.setOptions(*settings); return COASTER_SUCCESS; } catch (const CoasterError& err) { return coaster_error_rc(err); Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3950) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -42,13 +42,18 @@ // Opaque pointer types typedef struct coaster_client coaster_client; -typedef struct coaster_settings coaster_settings; +/* + Treat some types as direct pointers to real classes for C++, but opaque + pointers to dummy structs for C code to allow typechecking, but prevent + manipulation of the rbjects. + */ #ifdef __cplusplus -// Treat as direct pointer to class for C++ +typedef class Settings coaster_settings; typedef class Job coaster_job; #else -// Treat as opaque pointer to unimplemented struct for C +// Treat these types as opaque pointer to unimplemented struct for C +typedef struct coaster_settings_opaque_ coaster_settings; typedef struct coaster_job_opaque_ coaster_job; #endif From yadunandb at ci.uchicago.edu Mon Jun 23 15:38:17 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 23 Jun 2014 15:38:17 -0500 (CDT) Subject: [Swift-commit] r7925 - trunk/docs/userguide Message-ID: <20140623203817.567269CC8F@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-23 15:38:16 -0500 (Mon, 23 Jun 2014) New Revision: 7925 Modified: trunk/docs/userguide/configuration Log: Updated trunk docs to use taskWalltime instead of maxWalltime in the the swift.properties example. Modified: trunk/docs/userguide/configuration =================================================================== --- trunk/docs/userguide/configuration 2014-06-21 00:42:00 UTC (rev 7924) +++ trunk/docs/userguide/configuration 2014-06-23 20:38:16 UTC (rev 7925) @@ -282,11 +282,11 @@ ----- site.westmere.provider=local:slurm site.westmere.jobsPerNode=12 -site.westmere.maxWalltime=00:05:00 +site.westmere.taskWalltime=00:05:00 site.westmere.queue=westmere site.westmere.initialScore=10000 site.westmere.filesystem=local -site.westmere.workdir=/scratch/midway/davidkelly999 +site.westmere.workdir=/scratch/midway/$USER ----- However, you can also simplify this by grouping site properties together with @@ -296,7 +296,7 @@ site.westmere { provider=local:slurm jobsPerNode=12 - maxWalltime=00:05:00 + taskWalltime=00:05:00 queue=westmere initialScore=10000 filesystem=local From swift at ci.uchicago.edu Mon Jun 23 16:10:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 23 Jun 2014 16:10:03 -0500 (CDT) Subject: [Swift-commit] cog r3952 Message-ID: <20140623211003.A49B68D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3952 | timgarmstrong | 2014-06-23 16:05:18 -0500 (Mon, 23 Jun 2014) | 1 line Add in facility to store error info. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3951) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -25,6 +25,7 @@ #include #include +#include #include "CoasterClient.h" #include "CoasterError.h" @@ -36,7 +37,7 @@ using std::string; /* - Struct just wraps the object + Struct just wraps the objects */ struct coaster_client { CoasterLoop loop; @@ -46,27 +47,41 @@ Constructor: initialize loop then client */ coaster_client(string serviceURL) : - loop(), client(serviceURL, loop) { + loop(), client(serviceURL, loop) { } +}; - }; +struct coaster_err_info { + string msg; + + coaster_err_info(const string _msg): msg(msg) {} }; +static pthread_key_t err_key; +static pthread_once_t err_key_init = PTHREAD_ONCE_INIT; + +static coaster_rc coaster_return_error(coaster_rc code, + const string &msg); static coaster_rc coaster_error_rc(const CoasterError &err); static coaster_rc exception_rc(const std::exception &ex); +static void init_err_key(void); +static void cleanup_err_key(void *errinfo); +static void clear_err_info(void); // Check for error -// TODO: store error message somewhere instead of printing? #define COASTER_CONDITION(cond, err_rc, err_msg) { \ - if (!(cond)) { fprintf(stderr, (err_msg)); return (err_rc); }} + if (!(cond)) { coaster_return_error((err_rc), (err_msg)); }} +// TODO: it's bad that this might allocate memory +#define COASTER_CHECK_MALLOC(ptr) { \ + if ((ptr) == NULL) { \ + coaster_return_error(COASTER_ERROR_OOM, "out of memory"); }} + coaster_rc coaster_client_start(const char *service_url, size_t service_url_len, coaster_client **client) COASTERS_THROWS_NOTHING { try { *client = new coaster_client(string(service_url, service_url_len)); - if (!(*client)) { - return COASTER_ERROR_OOM; - } + COASTER_CHECK_MALLOC(*client); (*client)->client.start(); return COASTER_SUCCESS; @@ -98,9 +113,7 @@ COASTERS_THROWS_NOTHING { try { *settings = new Settings(); - if (!(*settings)) { - return COASTER_ERROR_OOM; - } + COASTER_CHECK_MALLOC(*settings); return COASTER_SUCCESS; } catch (const CoasterError& err) { @@ -159,15 +172,13 @@ // Use malloc so C client code can free *keys = (const char**)malloc(sizeof((*keys)[0]) * (*count)); - if (!(*keys)) { - return COASTER_ERROR_OOM; - } + COASTER_CHECK_MALLOC(*keys); if (key_lens != NULL) { *key_lens = (size_t *)malloc(sizeof((*key_lens)[0]) * (*count)); if (!(*key_lens)) { free(*keys); - return COASTER_ERROR_OOM; + COASTER_CHECK_MALLOC(*key_lens); } } @@ -205,7 +216,7 @@ coaster_settings *settings) COASTERS_THROWS_NOTHING { if (settings == NULL || client == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } try { @@ -259,7 +270,7 @@ const char *stderr_loc, size_t stderr_loc_len) COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } try { @@ -288,7 +299,7 @@ coaster_job_set_directory(coaster_job *job, const char *dir, size_t dir_len) COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, ""); } try { @@ -309,7 +320,7 @@ const char **names, size_t *name_lens, const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } try { @@ -342,7 +353,7 @@ const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); } try { @@ -371,7 +382,7 @@ COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); } try { @@ -403,7 +414,8 @@ COASTERS_THROWS_NOTHING { const JobStatus *status; if (job == NULL || (status = job->getStatus()) == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, + "invalid or unsubmitted job"); } *code = status->getStatusCode(); @@ -416,7 +428,7 @@ const char **stderr_s, size_t *stderr_len) COASTERS_THROWS_NOTHING { if (job == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); } const string *out = job->getStdout(); @@ -458,7 +470,7 @@ coaster_job **jobs, int *njobs) COASTERS_THROWS_NOTHING { if (client == NULL) { - return COASTER_ERROR_INVALID; + return coaster_return_error(COASTER_ERROR_INVALID, "invalid client"); } try { @@ -495,17 +507,82 @@ } } +const char *coaster_last_err_info(void) { + // Ensure initialized + pthread_once(&err_key_init, init_err_key); + + void *val = pthread_getspecific(err_key); + if (val == NULL) { + return NULL; + } + + coaster_err_info *info = static_cast(val); + return info->msg.c_str(); +} + +static void init_err_key(void) { + pthread_key_create(&err_key, cleanup_err_key); +} + /* + * Called to cleanup error key + */ +static void cleanup_err_key(void *errinfo) { + delete static_cast(errinfo); +} + +static void set_err_info(const string& msg) { + // Ensure key is initialized + pthread_once(&err_key_init, init_err_key); + + void *prev = pthread_getspecific(err_key); + if (prev != NULL) { + cleanup_err_key(prev); + } + + coaster_err_info *err_info = new coaster_err_info(msg); + pthread_setspecific(err_key, static_cast(err_info)); +} + +static void clear_err_info(void) { + void *prev = pthread_getspecific(err_key); + if (prev != NULL) { + cleanup_err_key(prev); + } + + pthread_setspecific(err_key, NULL); +} + +/* + * Helper to set error info when returning error. + */ +static coaster_rc coaster_return_error(coaster_rc code, + const string& msg) { + set_err_info(msg); + return code; +} + +/* * Set error information and return appropriate code */ static coaster_rc coaster_error_rc(const CoasterError &err) { - // TODO: store detailed error info + const char *msg = err.what(); + if (msg != NULL) { + set_err_info(string(msg)); + } else { + clear_err_info(); + } // TODO: distinguish different cases? return COASTER_ERROR_UNKNOWN; } static coaster_rc exception_rc(const std::exception &ex) { - // TODO: store error info? + const char *msg = ex.what(); + if (msg != NULL) { + set_err_info(string(msg)); + } else { + clear_err_info(); + } // TODO: handle specific types, e.g. bad_alloc return COASTER_ERROR_UNKNOWN; } Index: modules/provider-coaster-c-client/src/CoasterError.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterError.cpp (revision 3951) +++ modules/provider-coaster-c-client/src/CoasterError.cpp (working copy) @@ -9,11 +9,13 @@ #include CoasterError::CoasterError(string msg) { + // TODO: this stores pointer to string of unknown lifetime message = msg.c_str(); } CoasterError::CoasterError(const char* format, ...) { va_list args; + // TODO: this isn't freed char* buf = (char *) malloc(MAX_MSG_LEN + 1); va_start(args, format); @@ -23,6 +25,7 @@ } CoasterError::CoasterError(const stringstream* ss) { + // TODO: this stores pointer to string of unknown lifetime message = ss->str().c_str(); } Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3951) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -286,6 +286,15 @@ */ const char *coaster_rc_string(coaster_rc code); +/* + * Get additional information about last returned error in current + * thread. + * returns: error message, NULL if no information available. The + * returned string is valid until the next call by this thread + * to a coasters API function. + */ +const char *coaster_last_err_info(void); + #ifdef __cplusplus } // extern "C" #endif From tga at ci.uchicago.edu Tue Jun 24 10:06:27 2014 From: tga at ci.uchicago.edu (tga at ci.uchicago.edu) Date: Tue, 24 Jun 2014 10:06:27 -0500 (CDT) Subject: [Swift-commit] r7926 - trunk/bin Message-ID: <20140624150627.ED2A19CC8F@svn.ci.uchicago.edu> Author: tga Date: 2014-06-24 10:06:27 -0500 (Tue, 24 Jun 2014) New Revision: 7926 Modified: trunk/bin/start-coaster-service Log: Add usage message to start-coaster-service Modified: trunk/bin/start-coaster-service =================================================================== --- trunk/bin/start-coaster-service 2014-06-23 20:38:16 UTC (rev 7925) +++ trunk/bin/start-coaster-service 2014-06-24 15:06:27 UTC (rev 7926) @@ -8,6 +8,25 @@ exit 1 } +# Display usage message +usage() +{ + echo "Usage: $(basename $0) [OPTION]..." >&2 + echo "Options:" >&2 + echo "-conf \ + Coasters service configuration file (default: coaster-service.conf)" >&2 +} + +# Report a problem, then usage, then exit +usage_crash() +{ + MSG=$1 + echo ${MSG} >&2 + echo + usage + exit 1 +} + # Wait for a file to be created, up to given amount of time wait_for_file() { @@ -84,7 +103,7 @@ while [ $# -gt 0 ]; do case $1 in -conf) CMDLN_CONF=$2; shift 2;; - *) echo "Do not recognize command line option: $1" 1>&2; exit 1;; + *) echo "Do not recognize command line option: $1" 1>&2; usage; exit 1;; esac done @@ -102,7 +121,7 @@ elif [ -f "coaster-service.conf" ]; then CONFIG_FILE="coaster-service.conf" else - crash "Cannot find coaster-service.conf!" + usage_crash "Cannot find coaster-service.conf!" fi echo "Start-coaster-service..." From tga at ci.uchicago.edu Tue Jun 24 10:11:01 2014 From: tga at ci.uchicago.edu (tga at ci.uchicago.edu) Date: Tue, 24 Jun 2014 10:11:01 -0500 (CDT) Subject: [Swift-commit] r7927 - trunk/bin Message-ID: <20140624151101.4253B9CC8F@svn.ci.uchicago.edu> Author: tga Date: 2014-06-24 10:11:01 -0500 (Tue, 24 Jun 2014) New Revision: 7927 Modified: trunk/bin/start-coaster-service Log: Don't require gensites to be on path for start-coasters-service Modified: trunk/bin/start-coaster-service =================================================================== --- trunk/bin/start-coaster-service 2014-06-24 15:06:27 UTC (rev 7926) +++ trunk/bin/start-coaster-service 2014-06-24 15:11:01 UTC (rev 7927) @@ -209,9 +209,9 @@ echo Generating sites.xml if [ -f "gensites.template" ]; then - gensites $( cat gensites.template ) -p $CONFIG_FILE > sites.xml + $SWIFT_BIN/gensites $( cat gensites.template ) -p $CONFIG_FILE > sites.xml else - gensites persistent-coasters -p $CONFIG_FILE > sites.xml + $SWIFT_BIN/gensites persistent-coasters -p $CONFIG_FILE > sites.xml fi # Start workers From tga at ci.uchicago.edu Tue Jun 24 10:11:06 2014 From: tga at ci.uchicago.edu (tga at ci.uchicago.edu) Date: Tue, 24 Jun 2014 10:11:06 -0500 (CDT) Subject: [Swift-commit] r7928 - trunk/bin Message-ID: <20140624151106.577759CC8F@svn.ci.uchicago.edu> Author: tga Date: 2014-06-24 10:11:06 -0500 (Tue, 24 Jun 2014) New Revision: 7928 Modified: trunk/bin/stop-coaster-service Log: Add usage to stop-coaster-service Modified: trunk/bin/stop-coaster-service =================================================================== --- trunk/bin/stop-coaster-service 2014-06-24 15:11:01 UTC (rev 7927) +++ trunk/bin/stop-coaster-service 2014-06-24 15:11:06 UTC (rev 7928) @@ -8,6 +8,25 @@ exit 1 } +# Display usage message +usage() +{ + echo "Usage: $(basename $0) [OPTION]..." >&2 + echo "Options:" >&2 + echo "-conf \ + Coasters service configuration file (default: coaster-service.conf)" >&2 +} + +# Report a problem, then usage, then exit +usage_crash() +{ + MSG=$1 + echo ${MSG} >&2 + echo + usage + exit 1 +} + # Location of required files PID_FILE="$HOME/.swift/.coaster-service-pids" SWIFTVM_INSTANCES="$HOME/.swift/.swiftvm_instances" @@ -17,7 +36,7 @@ while [ $# -gt 0 ]; do case $1 in -conf) CMDLN_CONF=$2; shift 2;; - *) echo "Do not recognize command line option: $1" 1>&2; exit 1;; + *) echo "Do not recognize command line option: $1" 1>&2; usage; exit 1;; esac done @@ -31,7 +50,7 @@ elif [ -f "$(dirname $(readlink -f $0))/../etc/coaster-service.conf" ]; then CONFIG_FILE="$(dirname $(readlink -f $0))/../etc/coaster-service.conf" else - crash "Cannot find coaster-service.conf!" + usage_crash "Cannot find coaster-service.conf!" fi source "$CONFIG_FILE" From swift at ci.uchicago.edu Tue Jun 24 12:10:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 24 Jun 2014 12:10:03 -0500 (CDT) Subject: [Swift-commit] cog r3956 Message-ID: <20140624171003.9046B8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3956 | timgarmstrong | 2014-06-24 12:07:25 -0500 (Tue, 24 Jun 2014) | 1 line Add note about buffers ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/Buffer.h =================================================================== --- modules/provider-coaster-c-client/src/Buffer.h (revision 3955) +++ modules/provider-coaster-c-client/src/Buffer.h (working copy) @@ -14,6 +14,11 @@ using namespace std; +/* + * TODO: need documentation on behavior of buffers w.r.t memory lifespan. + * May need to disable copy constructors or add move constructors in + * some cases too. + */ class Buffer { protected: int len; From swift at ci.uchicago.edu Tue Jun 24 16:20:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 24 Jun 2014 16:20:03 -0500 (CDT) Subject: [Swift-commit] cog r3959 Message-ID: <20140624212003.9A0DD8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3959 | timgarmstrong | 2014-06-24 16:16:43 -0500 (Tue, 24 Jun 2014) | 1 line For some reason probably due to overload resolution this fixes log lock deadlock issue ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/Logger.cpp =================================================================== --- modules/provider-coaster-c-client/src/Logger.cpp (revision 3958) +++ modules/provider-coaster-c-client/src/Logger.cpp (working copy) @@ -52,7 +52,7 @@ } } -Logger& Logger::operator<< (string& str) { +Logger& Logger::operator<< (const string& str) { header(); if (level >= threshold) { buffer << str; Index: modules/provider-coaster-c-client/src/Lock.cpp =================================================================== --- modules/provider-coaster-c-client/src/Lock.cpp (revision 3958) +++ modules/provider-coaster-c-client/src/Lock.cpp (working copy) @@ -11,8 +11,8 @@ static int unique = 0; -#define DEBUG_LOCKS 1 -#define DEBUG_LOCKS_STACKS 1 +#define DEBUG_LOCKS 0 +#define DEBUG_LOCKS_STACKS 0 #if DEBUG_LOCKS_STACKS == 1 #include Index: modules/provider-coaster-c-client/src/Logger.h =================================================================== --- modules/provider-coaster-c-client/src/Logger.h (revision 3958) +++ modules/provider-coaster-c-client/src/Logger.h (working copy) @@ -41,7 +41,7 @@ public: virtual ~Logger(); Logger& operator<< (Level level); - Logger& operator<< (string& str); + Logger& operator<< (const string& str); Logger& operator<< (const string* str); Logger& operator<< (const char* str); Logger& operator<< (int i); From hategan at ci.uchicago.edu Tue Jun 24 16:31:30 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Tue, 24 Jun 2014 16:31:30 -0500 (CDT) Subject: [Swift-commit] r7929 - trunk/src/org/griphyn/vdl/util Message-ID: <20140624213130.92D339CC8F@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-24 16:31:30 -0500 (Tue, 24 Jun 2014) New Revision: 7929 Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java Log: set execution retries to 0 by default Modified: trunk/src/org/griphyn/vdl/util/VDL2Config.java =================================================================== --- trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-24 15:11:06 UTC (rev 7928) +++ trunk/src/org/griphyn/vdl/util/VDL2Config.java 2014-06-24 21:31:30 UTC (rev 7929) @@ -113,6 +113,8 @@ put(VDL2ConfigProperties.SITEDIR_KEEP, "false", ConfigPropertyType.BOOLEAN); put(VDL2ConfigProperties.PROVENANCE_LOG, "false", ConfigPropertyType.BOOLEAN); + put("execution.retries", "0", ConfigPropertyType.INT); + put("replication.enabled", "false", ConfigPropertyType.BOOLEAN); put("replication.min.queue.time", "60", ConfigPropertyType.INT); put("replication.limit", "3", ConfigPropertyType.INT); From swift at ci.uchicago.edu Tue Jun 24 23:05:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 24 Jun 2014 23:05:03 -0500 (CDT) Subject: [Swift-commit] cog r3966 Message-ID: <20140625040503.B80728D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3966 | timgarmstrong | 2014-06-24 23:02:21 -0500 (Tue, 24 Jun 2014) | 1 line Note on remaining memory leak ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterChannel.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.cpp (revision 3965) +++ modules/provider-coaster-c-client/src/CoasterChannel.cpp (working copy) @@ -54,10 +54,10 @@ CoasterChannel::~CoasterChannel() { delete rhdr.buf; - for (map::iterator it = handlers.begin(); - it != handlers.end(); ++it) { - delete it->second; - } + for (map::iterator it = handlers.begin(); + it != handlers.end(); ++it) { + delete it->second; + } } @@ -86,6 +86,7 @@ // full header read msg.reset(); decodeHeader(&rtag, &rflags, &rlen); + // TODO: this is being leaked msg.buf = new DynamicBuffer(rlen); readState = READ_STATE_DATA; } From swift at ci.uchicago.edu Tue Jun 24 23:50:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Tue, 24 Jun 2014 23:50:03 -0500 (CDT) Subject: [Swift-commit] cog r3967 Message-ID: <20140625045003.D58418D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3967 | hategan | 2014-06-24 23:47:52 -0500 (Tue, 24 Jun 2014) | 1 line unclevered statement ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/JobSubmitCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (revision 3966) +++ modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (working copy) @@ -99,14 +99,19 @@ ss.append(1, '='); while (*value) { char c = *value; - // TODO: are fallthroughs deliberate? + // TODO: are fallthroughs deliberate? Yes, they were. switch (c) { case '\n': - c = 'n'; + ss.append(1, '\\'); + ss.append(1, 'n'); + break; case '\\': ss.append(1, '\\'); + ss.append(1, '\\'); + break; default: ss.append(1, c); + break; } value++; n--; From swift at ci.uchicago.edu Wed Jun 25 11:15:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 25 Jun 2014 11:15:04 -0500 (CDT) Subject: [Swift-commit] cog r3972 Message-ID: <20140625161504.6038B8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3972 | timgarmstrong | 2014-06-25 11:10:35 -0500 (Wed, 25 Jun 2014) | 1 line Put all C++ classes into Coaster namespace. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterChannel.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.h (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterChannel.h (working copy) @@ -30,13 +30,14 @@ #include "Logger.h" #include "RemoteCoasterException.h" +namespace Coaster { + class Command; class Handler; class CoasterLoop; class CoasterClient; class HandlerFactory; - class DataChunk { private: /* Disable default copy constructor */ @@ -125,4 +126,6 @@ return os; } +} + #endif /* COASTER_CHANNEL_H_ */ Index: modules/provider-coaster-c-client/src/CoasterSWIG.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterSWIG.h (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterSWIG.h (working copy) @@ -12,6 +12,8 @@ %} #endif +namespace Coaster { + CoasterLoop* CoasterSWIGLoopCreate(void); int CoasterSWIGLoopDestroy(CoasterLoop* loop); @@ -35,3 +37,4 @@ //int CoasterSWIGTest(CoasterLoop *loop, char *service_URL); //int CoasterSWIGTest(CoasterClient *client); +} Index: modules/provider-coaster-c-client/src/CoasterLoop.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterLoop.h (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterLoop.h (working copy) @@ -19,8 +19,7 @@ #include #include -// 1 minute -#define HEARTBEAT_CHECK_INTERVAL 60 +namespace Coaster { class CoasterLoop { private: @@ -77,4 +76,5 @@ void checkHeartbeats(); }; +} #endif /* COASTER_LOOP_H_ */ Index: modules/provider-coaster-c-client/src/ChannelConfigurationCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/ChannelConfigurationCommand.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/ChannelConfigurationCommand.cpp (working copy) @@ -2,6 +2,8 @@ #include #include "Logger.h" +using namespace Coaster; + using std::string; using std::stringstream; Index: modules/provider-coaster-c-client/src/CoasterClient.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterClient.cpp (working copy) @@ -23,6 +23,8 @@ #include "Logger.h" +using namespace Coaster; + using std::list; using std::map; using std::string; Index: modules/provider-coaster-c-client/src/CoasterClientTest.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClientTest.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterClientTest.cpp (working copy) @@ -7,6 +7,8 @@ #include "Job.h" #include "Settings.h" +using namespace Coaster; + using std::cerr; using std::cout; using std::exception; Index: modules/provider-coaster-c-client/src/ConditionVariable.h =================================================================== --- modules/provider-coaster-c-client/src/ConditionVariable.h (revision 3971) +++ modules/provider-coaster-c-client/src/ConditionVariable.h (working copy) @@ -11,6 +11,8 @@ #include #include "Lock.h" +namespace Coaster { + class ConditionVariable { private: pthread_cond_t cv; @@ -27,4 +29,6 @@ void broadcast(); }; +} + #endif /* CONDITIONVARIABLE_H_ */ Index: modules/provider-coaster-c-client/src/CmdCBCV.cpp =================================================================== --- modules/provider-coaster-c-client/src/CmdCBCV.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CmdCBCV.cpp (working copy) @@ -7,6 +7,8 @@ #include "CmdCBCV.h" +using namespace Coaster; + using std::string; CmdCBCV::CmdCBCV() { Index: modules/provider-coaster-c-client/src/HeartBeatCommand.h =================================================================== --- modules/provider-coaster-c-client/src/HeartBeatCommand.h (revision 3971) +++ modules/provider-coaster-c-client/src/HeartBeatCommand.h (working copy) @@ -12,6 +12,8 @@ #include #include +namespace Coaster { + class HeartBeatCommand: public Command { private: long sendtime; @@ -24,4 +26,6 @@ virtual void replyReceived(); }; +} + #endif /* HEARTBEATCOMMAND_H_ */ Index: modules/provider-coaster-c-client/src/ClientHandlerFactory.cpp =================================================================== --- modules/provider-coaster-c-client/src/ClientHandlerFactory.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/ClientHandlerFactory.cpp (working copy) @@ -8,6 +8,8 @@ #include "ClientHandlerFactory.h" #include "JobStatusHandler.h" +using namespace Coaster; + ClientHandlerFactory::ClientHandlerFactory() { addHandler("JOBSTATUS"); } Index: modules/provider-coaster-c-client/src/RequestReply.cpp =================================================================== --- modules/provider-coaster-c-client/src/RequestReply.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/RequestReply.cpp (working copy) @@ -8,6 +8,8 @@ #include "RequestReply.h" #include "CoasterError.h" +using namespace Coaster; + #include using std::list; Index: modules/provider-coaster-c-client/src/Settings.h =================================================================== --- modules/provider-coaster-c-client/src/Settings.h (revision 3971) +++ modules/provider-coaster-c-client/src/Settings.h (working copy) @@ -11,6 +11,8 @@ #include #include +namespace Coaster { + class Settings { private: /* @@ -79,4 +81,5 @@ return os; } +} #endif /* SETTINGS_H_ */ Index: modules/provider-coaster-c-client/src/HandlerFactory.h =================================================================== --- modules/provider-coaster-c-client/src/HandlerFactory.h (revision 3971) +++ modules/provider-coaster-c-client/src/HandlerFactory.h (working copy) @@ -12,6 +12,8 @@ #include #include "Handler.h" +namespace Coaster { + class Handler; class HandlerFactory { @@ -31,5 +33,6 @@ creators[name] = &newHandler; } +} #endif /* HANDLERFACTORY_H_ */ Index: modules/provider-coaster-c-client/src/Buffer.h =================================================================== --- modules/provider-coaster-c-client/src/Buffer.h (revision 3971) +++ modules/provider-coaster-c-client/src/Buffer.h (working copy) @@ -12,6 +12,8 @@ #include #include +namespace Coaster { + /* * TODO: need documentation on behavior of buffers w.r.t memory lifespan. */ @@ -73,4 +75,6 @@ virtual char* getModifiableData(); }; +} + #endif /* BUFFER_H_ */ Index: modules/provider-coaster-c-client/src/JobStatus.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobStatus.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/JobStatus.cpp (working copy) @@ -9,6 +9,8 @@ #include #include "JobStatus.h" +using namespace Coaster; + using std::string; void JobStatus::init(JobStatusCode statusCode, time_t time, const string* message, RemoteCoasterException* exception) { @@ -82,7 +84,7 @@ } } -const char* statusCodeToStr(JobStatusCode code) { +const char* Coaster::statusCodeToStr(JobStatusCode code) { switch (code) { case UNSUBMITTED: return "UNSUBMITTED"; case SUBMITTING: return "SUBMITTING"; Index: modules/provider-coaster-c-client/src/Logger.cpp =================================================================== --- modules/provider-coaster-c-client/src/Logger.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Logger.cpp (working copy) @@ -10,6 +10,8 @@ #include #include +using namespace Coaster; + using std::cout; using std::ostream; using std::string; @@ -167,7 +169,7 @@ } } -Logger& endl(Logger& l) { +Logger& Coaster::endl(Logger& l) { l.endItem(); return l; } Index: modules/provider-coaster-c-client/src/Job.h =================================================================== --- modules/provider-coaster-c-client/src/Job.h (revision 3971) +++ modules/provider-coaster-c-client/src/Job.h (working copy) @@ -16,6 +16,8 @@ #include #include +namespace Coaster { + /* Job represents a single Job that is to be submitted to coasters. The Job object is created and has its parameters set before submission. @@ -135,4 +137,6 @@ const std::string* getStderr() const; }; +} + #endif /* JOB_DESCRIPTION_H_ */ Index: modules/provider-coaster-c-client/src/CoasterError.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterError.h (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterError.h (working copy) @@ -13,7 +13,7 @@ #include #include -#define MAX_MSG_LEN 256 +namespace Coaster { class CoasterError: public std::exception { private: @@ -26,4 +26,6 @@ virtual const char* what() const throw(); }; +} + #endif /* COASTERERROR_H_ */ Index: modules/provider-coaster-c-client/src/RunCoasterJob.cpp =================================================================== --- modules/provider-coaster-c-client/src/RunCoasterJob.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/RunCoasterJob.cpp (working copy) @@ -11,6 +11,8 @@ #include "Settings.h" #include "Logger.h" +using namespace Coaster; + using std::cerr; using std::cout; using std::exception; Index: modules/provider-coaster-c-client/src/JobStatusHandler.h =================================================================== --- modules/provider-coaster-c-client/src/JobStatusHandler.h (revision 3971) +++ modules/provider-coaster-c-client/src/JobStatusHandler.h (working copy) @@ -11,6 +11,8 @@ #include "Handler.h" #include "JobStatus.h" +namespace Coaster { + class JobStatusHandler: public Handler { public: JobStatusHandler(); @@ -18,4 +20,6 @@ virtual void requestReceived(); }; +} + #endif /* JOBSTATUSHANDLER_H_ */ Index: modules/provider-coaster-c-client/src/Handler.h =================================================================== --- modules/provider-coaster-c-client/src/Handler.h (revision 3971) +++ modules/provider-coaster-c-client/src/Handler.h (working copy) @@ -11,6 +11,8 @@ #include "CoasterChannel.h" #include "RequestReply.h" +namespace Coaster { + class CoasterChannel; class Handler: public RequestReply { @@ -29,4 +31,6 @@ virtual void dataSent(Buffer* buf); }; +} + #endif /* HANDLER_H_ */ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -30,7 +30,10 @@ #include "CoasterClient.h" #include "CoasterError.h" #include "CoasterLoop.h" +#include "Settings.h" +using namespace Coaster; + using std::malloc; using std::free; Index: modules/provider-coaster-c-client/src/Command.h =================================================================== --- modules/provider-coaster-c-client/src/Command.h (revision 3971) +++ modules/provider-coaster-c-client/src/Command.h (working copy) @@ -17,6 +17,8 @@ #include "CommandCallback.h" #include "RemoteCoasterException.h" +namespace Coaster { + class CoasterChannel; class CommandCallback; @@ -55,4 +57,6 @@ return os; } +} + #endif /* COMMAND_H_ */ Index: modules/provider-coaster-c-client/src/ChannelCallback.cpp =================================================================== --- modules/provider-coaster-c-client/src/ChannelCallback.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/ChannelCallback.cpp (working copy) @@ -7,6 +7,8 @@ #include "ChannelCallback.h" +using namespace Coaster; + ChannelCallback::ChannelCallback() { } Index: modules/provider-coaster-c-client/src/Lock.cpp =================================================================== --- modules/provider-coaster-c-client/src/Lock.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Lock.cpp (working copy) @@ -9,6 +9,8 @@ #include "Lock.h" +using namespace Coaster; + static int unique = 0; #define DEBUG_LOCKS 0 Index: modules/provider-coaster-c-client/src/CoasterChannel.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterChannel.cpp (working copy) @@ -18,6 +18,8 @@ #include #include +using namespace Coaster; + using std::exception; using std::list; using std::map; Index: modules/provider-coaster-c-client/src/CoasterLoop.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterLoop.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterLoop.cpp (working copy) @@ -13,10 +13,15 @@ #include #include +using namespace Coaster; + using std::list; using std::map; using std::string; +// 1 minute +#define HEARTBEAT_CHECK_INTERVAL 60 + void* run(void* ptr); void checkSelectError(int ret); Index: modules/provider-coaster-c-client/src/CoasterSWIG.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterSWIG.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterSWIG.cpp (working copy) @@ -11,6 +11,8 @@ #include "Settings.h" #include "Job.h" +using namespace Coaster; + /** CoasterSWIGLoopCreate : create, starts and returns * a pointer to a CoasterLoop object. */ Index: modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h =================================================================== --- modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h (revision 3971) +++ modules/provider-coaster-c-client/src/ServiceConfigurationCommand.h (working copy) @@ -7,6 +7,8 @@ #include "Buffer.h" #include "Settings.h" +namespace Coaster { + class ServiceConfigurationCommand: public Command { private: Settings* settings; @@ -19,4 +21,6 @@ void serialize(); }; +} + #endif Index: modules/provider-coaster-c-client/src/ConditionVariable.cpp =================================================================== --- modules/provider-coaster-c-client/src/ConditionVariable.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/ConditionVariable.cpp (working copy) @@ -7,6 +7,8 @@ #include "ConditionVariable.h" +using namespace Coaster; + ConditionVariable::ConditionVariable() { pthread_cond_init(&cv, NULL); } Index: modules/provider-coaster-c-client/src/HeartBeatCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/HeartBeatCommand.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/HeartBeatCommand.cpp (working copy) @@ -9,6 +9,8 @@ #include "Logger.h" #include +using namespace Coaster; + using std::string; string HeartBeatCommand::NAME("HEARTBEAT"); Index: modules/provider-coaster-c-client/src/StagingSetEntry.h =================================================================== --- modules/provider-coaster-c-client/src/StagingSetEntry.h (revision 3971) +++ modules/provider-coaster-c-client/src/StagingSetEntry.h (working copy) @@ -10,6 +10,8 @@ #include +namespace Coaster { + enum StagingMode { ALWAYS = 1, IF_PRESENT = 2, ON_ERROR = 4, ON_SUCCESS = 8 }; class StagingSetEntry { @@ -25,4 +27,6 @@ virtual ~StagingSetEntry(); }; +} + #endif /* STAGINGSETENTRY_H_ */ Index: modules/provider-coaster-c-client/src/CommandCallback.h =================================================================== --- modules/provider-coaster-c-client/src/CommandCallback.h (revision 3971) +++ modules/provider-coaster-c-client/src/CommandCallback.h (working copy) @@ -11,6 +11,8 @@ #include #include "RemoteCoasterException.h" +namespace Coaster { + class Command; class CommandCallback { @@ -19,5 +21,6 @@ virtual void replyReceived(Command* cmd) = 0; }; +} #endif /* COMMANDCALLBACK_H_ */ Index: modules/provider-coaster-c-client/src/RemoteCoasterException.h =================================================================== --- modules/provider-coaster-c-client/src/RemoteCoasterException.h (revision 3971) +++ modules/provider-coaster-c-client/src/RemoteCoasterException.h (working copy) @@ -10,6 +10,8 @@ #include +namespace Coaster { + class RemoteCoasterException { private: std::string* className; @@ -20,4 +22,6 @@ std::string& str(); }; +} + #endif /* REMOTECOASTEREXCEPTION_H_ */ Index: modules/provider-coaster-c-client/src/Settings.cpp =================================================================== --- modules/provider-coaster-c-client/src/Settings.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Settings.cpp (working copy) @@ -7,6 +7,8 @@ #include "Settings.h" +using namespace Coaster; + using std::string; using std::map; Index: modules/provider-coaster-c-client/src/JobSubmitCommand.h =================================================================== --- modules/provider-coaster-c-client/src/JobSubmitCommand.h (revision 3971) +++ modules/provider-coaster-c-client/src/JobSubmitCommand.h (working copy) @@ -7,6 +7,8 @@ #include #include "Buffer.h" +namespace Coaster { + class JobSubmitCommand: public Command { private: Job* job; @@ -21,4 +23,6 @@ void serialize(); }; +} + #endif Index: modules/provider-coaster-c-client/src/ChannelConfigurationCommand.h =================================================================== --- modules/provider-coaster-c-client/src/ChannelConfigurationCommand.h (revision 3971) +++ modules/provider-coaster-c-client/src/ChannelConfigurationCommand.h (working copy) @@ -6,6 +6,8 @@ #include #include "Buffer.h" +namespace Coaster { + class ChannelConfigurationCommand: public Command { private: std::string localId; @@ -21,4 +23,6 @@ void serialize(); }; +} + #endif Index: modules/provider-coaster-c-client/src/CoasterClient.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.h (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterClient.h (working copy) @@ -26,6 +26,8 @@ #endif +namespace Coaster { + class ClientHandlerFactory; class HandlerFactory; class CoasterLoop; @@ -112,4 +114,6 @@ void updateJobStatusNoLock(Job* job, JobStatus* status); }; +} + #endif /* COASTER_CLIENT_H_ */ Index: modules/provider-coaster-c-client/src/HandlerFactory.cpp =================================================================== --- modules/provider-coaster-c-client/src/HandlerFactory.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/HandlerFactory.cpp (working copy) @@ -7,6 +7,8 @@ #include "HandlerFactory.h" +using namespace Coaster; + using std::string; using std::map; Index: modules/provider-coaster-c-client/src/CmdCBCV.h =================================================================== --- modules/provider-coaster-c-client/src/CmdCBCV.h (revision 3971) +++ modules/provider-coaster-c-client/src/CmdCBCV.h (working copy) @@ -13,6 +13,8 @@ #include "ConditionVariable.h" #include "RemoteCoasterException.h" +namespace Coaster { + class CmdCBCV: public CommandCallback { private: bool done; @@ -26,4 +28,5 @@ void wait(); }; +} #endif /* CMDCBCV_H_ */ Index: modules/provider-coaster-c-client/src/Buffer.cpp =================================================================== --- modules/provider-coaster-c-client/src/Buffer.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Buffer.cpp (working copy) @@ -9,6 +9,8 @@ #include #include +using namespace Coaster; + using std::string; using std::out_of_range; using std::logic_error; Index: modules/provider-coaster-c-client/src/ClientHandlerFactory.h =================================================================== --- modules/provider-coaster-c-client/src/ClientHandlerFactory.h (revision 3971) +++ modules/provider-coaster-c-client/src/ClientHandlerFactory.h (working copy) @@ -10,10 +10,14 @@ #include "HandlerFactory.h" +namespace Coaster { + class ClientHandlerFactory: public HandlerFactory { public: ClientHandlerFactory(); virtual ~ClientHandlerFactory(); }; +} + #endif /* CLIENTHANDLERFACTORY_H_ */ Index: modules/provider-coaster-c-client/src/RequestReply.h =================================================================== --- modules/provider-coaster-c-client/src/RequestReply.h (revision 3971) +++ modules/provider-coaster-c-client/src/RequestReply.h (working copy) @@ -14,6 +14,8 @@ #include #include +namespace Coaster { + class CoasterChannel; class RequestReply: public ChannelCallback { @@ -61,4 +63,6 @@ virtual void receiveCompleted(int flags) = 0; }; +} + #endif /* REQUESTREPLY_H_ */ Index: modules/provider-coaster-c-client/src/CoasterError.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterError.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/CoasterError.cpp (working copy) @@ -8,9 +8,13 @@ #include "CoasterError.h" #include +using namespace Coaster; + using std::string; using std::stringstream; +#define MAX_MSG_LEN 256 + CoasterError::CoasterError(const string& msg) { this->message = msg; } Index: modules/provider-coaster-c-client/src/Job.cpp =================================================================== --- modules/provider-coaster-c-client/src/Job.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Job.cpp (working copy) @@ -2,6 +2,8 @@ #include #include +using namespace Coaster; + using std::map; using std::pair; using std::string; Index: modules/provider-coaster-c-client/src/JobStatusHandler.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobStatusHandler.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/JobStatusHandler.cpp (working copy) @@ -9,6 +9,8 @@ #include +using namespace Coaster; + using std::string; using std::stringstream; Index: modules/provider-coaster-c-client/src/Handler.cpp =================================================================== --- modules/provider-coaster-c-client/src/Handler.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Handler.cpp (working copy) @@ -2,6 +2,8 @@ #include "Logger.h" +using namespace Coaster; + using std::list; using std::string; using std::vector; Index: modules/provider-coaster-c-client/src/Command.cpp =================================================================== --- modules/provider-coaster-c-client/src/Command.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/Command.cpp (working copy) @@ -7,6 +7,8 @@ using std::string; using std::vector; +using namespace Coaster; + Command::Command(const string* pname) { name = pname; ferrorReceived = false; Index: modules/provider-coaster-c-client/src/JobStatus.h =================================================================== --- modules/provider-coaster-c-client/src/JobStatus.h (revision 3971) +++ modules/provider-coaster-c-client/src/JobStatus.h (working copy) @@ -13,6 +13,8 @@ #include "RemoteCoasterException.h" #include "coaster-defs.h" +namespace Coaster { + class JobStatus { private: JobStatusCode statusCode; @@ -62,4 +64,6 @@ return os; } +} + #endif /* JOB_STATUS_H_ */ Index: modules/provider-coaster-c-client/src/Logger.h =================================================================== --- modules/provider-coaster-c-client/src/Logger.h (revision 3971) +++ modules/provider-coaster-c-client/src/Logger.h (working copy) @@ -16,6 +16,7 @@ #include #include "Lock.h" +namespace Coaster { class Logger: public std::ostream { public: @@ -74,4 +75,6 @@ #define LogInfo Logger::singleton().setFile(__FILE__) << Logger::INFO #define LogDebug Logger::singleton().setFile(__FILE__) << Logger::DEBUG +} + #endif /* LOGGER_H_ */ Index: modules/provider-coaster-c-client/src/ServiceConfigurationCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/ServiceConfigurationCommand.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/ServiceConfigurationCommand.cpp (working copy) @@ -1,6 +1,8 @@ #include "ServiceConfigurationCommand.h" #include +using namespace Coaster; + using std::map; using std::string; Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3971) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -49,8 +49,13 @@ manipulation of the rbjects. */ #ifdef __cplusplus -typedef class Settings coaster_settings; -typedef class Job coaster_job; +namespace Coaster { + class Settings; + class Job; +} + +typedef class Coaster::Settings coaster_settings; +typedef class Coaster::Job coaster_job; #else // Treat these types as opaque pointer to unimplemented struct for C typedef struct coaster_settings_opaque_ coaster_settings; Index: modules/provider-coaster-c-client/src/ChannelCallback.h =================================================================== --- modules/provider-coaster-c-client/src/ChannelCallback.h (revision 3971) +++ modules/provider-coaster-c-client/src/ChannelCallback.h (working copy) @@ -10,6 +10,8 @@ #include "Buffer.h" +namespace Coaster { + class ChannelCallback { public: ChannelCallback(); @@ -23,4 +25,6 @@ virtual void dataSent(Buffer* buf); }; +} + #endif /* CHANNELCALLBACK_H_ */ Index: modules/provider-coaster-c-client/src/StagingSetEntry.cpp =================================================================== --- modules/provider-coaster-c-client/src/StagingSetEntry.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/StagingSetEntry.cpp (working copy) @@ -7,6 +7,8 @@ #include "StagingSetEntry.h" +using namespace Coaster; + using std::string; StagingSetEntry::StagingSetEntry(string psource, string pdestination, StagingMode pmode) { Index: modules/provider-coaster-c-client/src/RemoteCoasterException.cpp =================================================================== --- modules/provider-coaster-c-client/src/RemoteCoasterException.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/RemoteCoasterException.cpp (working copy) @@ -9,6 +9,8 @@ #include #include +using namespace Coaster; + using std::exception; using std::string; Index: modules/provider-coaster-c-client/src/JobSubmitCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (revision 3971) +++ modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (working copy) @@ -3,6 +3,8 @@ #include #include +using namespace Coaster; + using std::cout; using std::map; using std::ostream; Index: modules/provider-coaster-c-client/src/Lock.h =================================================================== --- modules/provider-coaster-c-client/src/Lock.h (revision 3971) +++ modules/provider-coaster-c-client/src/Lock.h (working copy) @@ -10,6 +10,8 @@ #include +namespace Coaster { + class Lock { int id; private: @@ -42,4 +44,6 @@ }; }; +} + #endif /* LOCK_H_ */ From swift at ci.uchicago.edu Wed Jun 25 12:30:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 25 Jun 2014 12:30:04 -0500 (CDT) Subject: [Swift-commit] cog r3976 Message-ID: <20140625173004.2349A8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3976 | timgarmstrong | 2014-06-25 12:25:55 -0500 (Wed, 25 Jun 2014) | 1 line Don't need TODO ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3975) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -98,8 +98,6 @@ coaster_rc coaster_client_stop(coaster_client *client) COASTERS_THROWS_NOTHING { try { - // TODO: stop things - client->client.stop(); client->loop.stop(); From swift at ci.uchicago.edu Wed Jun 25 17:20:32 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 25 Jun 2014 17:20:32 -0500 (CDT) Subject: [Swift-commit] cog r3981 Message-ID: <20140625222032.D15528D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3981 | timgarmstrong | 2014-06-25 17:18:13 -0500 (Wed, 25 Jun 2014) | 1 line Add ability to get string representing job. Slightly simplify code by removing pointer indirection. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/Job.h =================================================================== --- modules/provider-coaster-c-client/src/Job.h (revision 3980) +++ modules/provider-coaster-c-client/src/Job.h (working copy) @@ -35,7 +35,7 @@ * it work to just store them by value and have zero-length * be equivalent to NULL. Are zero-length strings meaningful? */ - std::vector* arguments; + std::vector arguments; std::string* directory; std::string* stdinLocation; std::string* stdoutLocation; @@ -55,7 +55,7 @@ JobStatus* status; - /* Disable default copy constructor */ + /* Disable default copy constructor */ Job(const Job&); /* Disable default assignment */ Job& operator=(const Job&); @@ -81,11 +81,11 @@ const std::string* getRemoteIdentity() const; void setRemoteIdentity(const std::string& remoteId); - std::vector* getArguments(); + const std::vector& getArguments(); - /* - * Add argument, taking ownership - */ + /* + * Add argument, taking ownership + */ void addArgument(std::string* arg); void addArgument(const std::string& arg); void addArgument(const char* arg); @@ -135,6 +135,12 @@ const std::string* getStdout() const; const std::string* getStderr() const; + + /* + * Return a human-readable string representation + * of the job. + */ + std::string toString() const; }; } Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3980) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -25,6 +25,7 @@ #include #include +#include #include #include "CoasterClient.h" @@ -38,6 +39,7 @@ using std::free; using std::string; +using std::memcpy; /* Struct just wraps the objects @@ -265,6 +267,29 @@ } coaster_rc +coaster_job_to_string(const coaster_job *job, char **str, size_t *str_len) + COASTERS_THROWS_NOTHING { + if (job == NULL || str == NULL || str_len == NULL) { + return coaster_return_error(COASTER_ERROR_INVALID, "invalid argument"); + } + + try { + string jobStr = job->toString(); + + *str = (char*)malloc(jobStr.length() + 1); + COASTER_CHECK_MALLOC(*str); + memcpy(*str, jobStr.c_str(), jobStr.length() + 1); + *str_len = jobStr.length(); + return COASTER_SUCCESS; + } catch (const CoasterError& err) { + return coaster_error_rc(err); + } catch (const std::exception& ex) { + return exception_rc(ex); + } + +} + +coaster_rc coaster_job_set_redirects(coaster_job *job, const char *stdin_loc, size_t stdin_loc_len, const char *stdout_loc, size_t stdout_loc_len, Index: modules/provider-coaster-c-client/src/Job.cpp =================================================================== --- modules/provider-coaster-c-client/src/Job.cpp (revision 3980) +++ modules/provider-coaster-c-client/src/Job.cpp (working copy) @@ -7,6 +7,7 @@ using std::map; using std::pair; using std::string; +using std::stringstream; using std::vector; static job_id_t seq = 0; @@ -15,7 +16,6 @@ executable = pexecutable; identity = seq++; - arguments = NULL; directory = NULL; stdinLocation = NULL; stdoutLocation = NULL; @@ -51,15 +51,12 @@ remoteIdentity = new string(remoteId); } -vector* Job::getArguments() { +const vector& Job::getArguments() { return arguments; } void Job::addArgument(string* arg) { - if (arguments == NULL) { - arguments = new vector; - } - arguments->push_back(arg); + arguments.push_back(arg); } const string& Job::getExecutable() const { @@ -250,6 +247,24 @@ return stderr; } +/* + * Just include the executable and arguments for now + */ +string Job::toString() const { + stringstream ss; + ss << executable; + for (vector::const_iterator it = arguments.begin(); + it != arguments.end(); ++it) { + const string *arg = *it; + if (arg == NULL) { + ss << " NULL"; + } else { + ss << " " << *arg ; + } + } + return ss.str(); +} + void Job::setStatus(JobStatus* newStatus) { // Since the client can process a job status while another // status is coming in, a status cannot be deleted when a new status comes in. @@ -276,10 +291,7 @@ if (remoteIdentity != NULL) { delete remoteIdentity; } - if (arguments != NULL) { - for (int i = 0; i < arguments->size(); i++) { - delete arguments->at(i); - } - delete arguments; + for (int i = 0; i < arguments.size(); i++) { + delete arguments.at(i); } } Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3980) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -191,6 +191,14 @@ coaster_job_free(coaster_job *job) COASTERS_THROWS_NOTHING; /* + * Create a human readable string describing job. + * str: output for dynamically allocated string, to be freed by caller + */ +coaster_rc +coaster_job_to_string(const coaster_job *job, char **str, size_t *str_len) + COASTERS_THROWS_NOTHING; + +/* * Set input and output stream redirections. * If set to NULL, don't modify. */ Index: modules/provider-coaster-c-client/src/JobSubmitCommand.cpp =================================================================== --- modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (revision 3980) +++ modules/provider-coaster-c-client/src/JobSubmitCommand.cpp (working copy) @@ -53,11 +53,9 @@ add(ss, "stderr", job->getStderrLocation()); - vector* arguments = job->getArguments(); - if (arguments != NULL) { - for (vector::iterator i = arguments->begin(); i != arguments->end(); ++i) { - add(ss, "arg", *i); - } + const vector& arguments = job->getArguments(); + for (vector::const_iterator i = arguments.begin(); i != arguments.end(); ++i) { + add(ss, "arg", *i); } map* env = job->getEnv(); From swift at ci.uchicago.edu Wed Jun 25 17:25:26 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 25 Jun 2014 17:25:26 -0500 (CDT) Subject: [Swift-commit] cog r3982 Message-ID: <20140625222526.99E4C8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3982 | timgarmstrong | 2014-06-25 17:25:04 -0500 (Wed, 25 Jun 2014) | 1 line Add const attribute where appropriate ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3981) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -430,13 +430,13 @@ } job_id_t -coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING { +coaster_job_get_id(const coaster_job *job) COASTERS_THROWS_NOTHING { // Shouldn't throw anything from accessor method return job->getIdentity(); } coaster_rc -coaster_job_status_code(coaster_job *job, coaster_job_status *code) +coaster_job_status_code(const coaster_job *job, coaster_job_status *code) COASTERS_THROWS_NOTHING { const JobStatus *status; if (job == NULL || (status = job->getStatus()) == NULL) { @@ -449,7 +449,7 @@ } coaster_rc -coaster_job_get_outstreams(coaster_job *job, +coaster_job_get_outstreams(const coaster_job *job, const char **stdout_s, size_t *stdout_len, const char **stderr_s, size_t *stderr_len) COASTERS_THROWS_NOTHING { Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3981) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -252,14 +252,14 @@ * a coasters job that is assigned when the job is created. */ job_id_t -coaster_job_get_id(coaster_job *job) COASTERS_THROWS_NOTHING; +coaster_job_get_id(const coaster_job *job) COASTERS_THROWS_NOTHING; /* * Get status of a submitted job. * Return COASTER_ERROR_INVALID if job is invalid or has no status. */ coaster_rc -coaster_job_status_code(coaster_job *job, coaster_job_status *code) +coaster_job_status_code(const coaster_job *job, coaster_job_status *code) COASTERS_THROWS_NOTHING; /* @@ -268,7 +268,7 @@ * are owned by job: will become invalid if job is freed. */ coaster_rc -coaster_job_get_outstreams(coaster_job *job, +coaster_job_get_outstreams(const coaster_job *job, const char **stdout_s, size_t *stdout_len, const char **stderr_s, size_t *stderr_len) COASTERS_THROWS_NOTHING; From swift at ci.uchicago.edu Wed Jun 25 21:15:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Wed, 25 Jun 2014 21:15:03 -0500 (CDT) Subject: [Swift-commit] cog r3983 Message-ID: <20140626021503.8F54B8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3983 | timgarmstrong | 2014-06-25 21:12:03 -0500 (Wed, 25 Jun 2014) | 1 line Fix mismatched function names ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3982) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -127,7 +127,7 @@ } coaster_rc coaster_settings_parse(coaster_settings *settings, - const char *str) + const char *str, size_t str_len) COASTERS_THROWS_NOTHING { // TODO: parsing using code currently in CoasterSwig From swift at ci.uchicago.edu Thu Jun 26 10:55:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Thu, 26 Jun 2014 10:55:03 -0500 (CDT) Subject: [Swift-commit] cog r3984 Message-ID: <20140626155503.C936A8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3984 | timgarmstrong | 2014-06-26 10:50:05 -0500 (Thu, 26 Jun 2014) | 1 line Start loop before starting client. Add checks, warning and documentation for correct loop start/stop. ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterLoop.h =================================================================== --- modules/provider-coaster-c-client/src/CoasterLoop.h (revision 3983) +++ modules/provider-coaster-c-client/src/CoasterLoop.h (working copy) @@ -65,6 +65,7 @@ * Add a channel for the loop to monitor. * Ownership of the channel is shared between caller and the loop. * Must be removed later by a call to removeChannel(). + * Loop must be started before adding channel */ void addChannel(CoasterChannel* channel); Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3983) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -88,6 +88,7 @@ *client = new coaster_client(string(service_url, service_url_len)); COASTER_CHECK_MALLOC(*client); + (*client)->loop.start(); (*client)->client.start(); return COASTER_SUCCESS; } catch (const CoasterError& err) { Index: modules/provider-coaster-c-client/src/CoasterLoop.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterLoop.cpp (revision 3983) +++ modules/provider-coaster-c-client/src/CoasterLoop.cpp (working copy) @@ -79,6 +79,8 @@ if (pthread_join(thread, NULL) != 0) { throw CoasterError(string("Could not join thread: ") + strerror(errno)); } + + // TODO: close pipe? LogInfo << "Coaster loop stopped" << endl; } @@ -214,15 +216,24 @@ } void CoasterLoop::addChannel(CoasterChannel* channel) { Lock::Scoped l(lock); + if (!started || done) { + LogWarn << "Add channel to non-running loop" << endl; + } addList.push_back(channel); } void CoasterLoop::removeChannel(CoasterChannel* channel, bool deleteChan) { Lock::Scoped l(lock); + if (!started || done) { + LogWarn << "Remove channel to non-running loop" << endl; + } removeList.push_back(pair(channel, deleteChan)); } void CoasterLoop::requestWrite(int count) { + if (!started) { + throw new CoasterError("requestWrite() on non-started loop"); + } writesPending += count; LogDebug << "request " << count << " writes; writesPending: " << writesPending << endl; char tmp[count]; From swift at ci.uchicago.edu Thu Jun 26 13:20:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Thu, 26 Jun 2014 13:20:04 -0500 (CDT) Subject: [Swift-commit] cog r3988 Message-ID: <20140626182004.5349B8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3988 | timgarmstrong | 2014-06-26 13:18:10 -0500 (Thu, 26 Jun 2014) | 1 line Add parentheses to comparison as suggested by warning ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterClient.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClient.cpp (revision 3987) +++ modules/provider-coaster-c-client/src/CoasterClient.cpp (working copy) @@ -94,7 +94,7 @@ // configure non-blocking int flags = fcntl(sockFD, F_GETFL); - if (flags & O_NONBLOCK == 0) { + if ((flags & O_NONBLOCK) == 0) { if (fcntl(sockFD, F_SETFL, flags | O_NONBLOCK) < 0) { throw CoasterError("Failed to configure socket for non-blocking operations: %s", strerror(errno)); } Index: modules/provider-coaster-c-client/src/CoasterClientTest.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterClientTest.cpp (revision 3987) +++ modules/provider-coaster-c-client/src/CoasterClientTest.cpp (working copy) @@ -11,6 +11,7 @@ using std::cerr; using std::cout; +using std::endl; using std::exception; using std::list; From swift at ci.uchicago.edu Thu Jun 26 15:00:04 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Thu, 26 Jun 2014 15:00:04 -0500 (CDT) Subject: [Swift-commit] cog r3989 Message-ID: <20140626200004.E370E8D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3989 | timgarmstrong | 2014-06-26 14:57:43 -0500 (Thu, 26 Jun 2014) | 1 line Support logging from pure C interface ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/Logger.cpp =================================================================== --- modules/provider-coaster-c-client/src/Logger.cpp (revision 3988) +++ modules/provider-coaster-c-client/src/Logger.cpp (working copy) @@ -18,8 +18,8 @@ Logger::Logger(ostream& pout) { out = &pout; - level = NONE; - threshold = INFO; // Only show info and higher by default + level = COASTER_LOG_NONE; + threshold = COASTER_LOG_INFO; // Only show info and higher by default file = ""; startOfItem = true; } @@ -27,32 +27,32 @@ Logger::~Logger() { } -Logger& Logger::operator<< (Level plevel) { +Logger& Logger::operator<< (CoasterLogLevel plevel) { setLevel(plevel); return *this; } -void Logger::setLevel(Level plevel) { +void Logger::setLevel(CoasterLogLevel plevel) { level = plevel; strLevel = levelToStr(level); } -void Logger::setThreshold(Level plevel) { +void Logger::setThreshold(CoasterLogLevel plevel) { threshold = plevel; } -const char* Logger::levelToStr(Level level) { +const char* Logger::levelToStr(CoasterLogLevel level) { switch (level) { - case NONE: + case COASTER_LOG_NONE: return "NONE "; - case ERROR: + case COASTER_LOG_ERROR: return "ERROR"; - case WARN: + case COASTER_LOG_WARN: return "WARN "; - case INFO: + case COASTER_LOG_INFO: return "INFO "; - case DEBUG: + case COASTER_LOG_DEBUG: return "DEBUG"; default: return "?????"; @@ -113,7 +113,7 @@ return *this; } -void Logger::log(Level level, const char* fileName, const char* msg) { +void Logger::log(CoasterLogLevel level, const char* fileName, const char* msg) { if (level >= threshold) { setLevel(level); setFile(fileName); @@ -122,7 +122,7 @@ commitBuffer(); } } -void Logger::log(Level level, const char* fileName, string msg) { +void Logger::log(CoasterLogLevel level, const char* fileName, string msg) { if (level >= threshold) { setLevel(level); setFile(fileName); Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3988) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -615,3 +615,10 @@ // TODO: handle specific types, e.g. bad_alloc return COASTER_ERROR_UNKNOWN; } + +coaster_rc coaster_set_log_threshold(coaster_log_level threshold) { + + Logger::singleton().setThreshold(threshold); + + return COASTER_SUCCESS; +} Index: modules/provider-coaster-c-client/src/Logger.h =================================================================== --- modules/provider-coaster-c-client/src/Logger.h (revision 3988) +++ modules/provider-coaster-c-client/src/Logger.h (working copy) @@ -14,38 +14,46 @@ #include #include #include + +#include "coaster-defs.h" #include "Lock.h" namespace Coaster { class Logger: public std::ostream { public: - enum Level { NONE = -1, DEBUG = 0, INFO = 1, WARN = 2, ERROR = 3, FATAL = 4 }; static Logger* instance; + + static const CoasterLogLevel NONE = COASTER_LOG_NONE; + static const CoasterLogLevel DEBUG = COASTER_LOG_DEBUG; + static const CoasterLogLevel INFO = COASTER_LOG_INFO; + static const CoasterLogLevel WARN = COASTER_LOG_WARN; + static const CoasterLogLevel ERROR = COASTER_LOG_ERROR; + static const CoasterLogLevel FATAL = COASTER_LOG_FATAL; private: std::ostream* out; std::stringstream buffer; - Level level, threshold; + CoasterLogLevel level, threshold; const char* file; const char* strLevel; char ts[TS_LEN + 1]; bool startOfItem; Lock lock; - /* Disable default copy constructor */ + /* Disable default copy constructor */ Logger(const Logger&); /* Disable default assignment */ Logger& operator=(const Logger&); protected: Logger(std::ostream& out); - void setLevel(Level level); - const char* levelToStr(Level level); + void setLevel(CoasterLogLevel level); + const char* levelToStr(CoasterLogLevel level); void header(); char* timeStamp(); void commitBuffer(); public: virtual ~Logger(); - Logger& operator<< (Level level); + Logger& operator<< (CoasterLogLevel level); Logger& operator<< (const std::string& str); Logger& operator<< (const std::string* str); Logger& operator<< (const char* str); @@ -54,10 +62,10 @@ Logger& operator<< (Logger& ( *pf )(Logger&)); Logger& setFile(const char* file); void endItem(); - void log(Level level, const char* fileName, const char* msg); - void log(Level level, const char* fileName, std::string msg); + void log(CoasterLogLevel level, const char* fileName, const char* msg); + void log(CoasterLogLevel level, const char* fileName, std::string msg); - void setThreshold(Level level); + void setThreshold(CoasterLogLevel level); static Logger& singleton(); }; Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3988) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -311,6 +311,8 @@ */ const char *coaster_last_err_info(void); +coaster_rc coaster_set_log_threshold(coaster_log_level threshold); + #ifdef __cplusplus } // extern "C" #endif From swift at ci.uchicago.edu Thu Jun 26 15:55:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Thu, 26 Jun 2014 15:55:03 -0500 (CDT) Subject: [Swift-commit] cog r3992 Message-ID: <20140626205503.82D438D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3992 | timgarmstrong | 2014-06-26 15:54:32 -0500 (Thu, 26 Jun 2014) | 1 line COASTER instead of COASTERS ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/coasters.cpp =================================================================== --- modules/provider-coaster-c-client/src/coasters.cpp (revision 3991) +++ modules/provider-coaster-c-client/src/coasters.cpp (working copy) @@ -83,7 +83,7 @@ coaster_rc coaster_client_start(const char *service_url, size_t service_url_len, - coaster_client **client) COASTERS_THROWS_NOTHING { + coaster_client **client) COASTER_THROWS_NOTHING { try { *client = new coaster_client(string(service_url, service_url_len)); COASTER_CHECK_MALLOC(*client); @@ -99,7 +99,7 @@ } coaster_rc coaster_client_stop(coaster_client *client) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { try { client->client.stop(); client->loop.stop(); @@ -114,7 +114,7 @@ } coaster_rc coaster_settings_create(coaster_settings **settings) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { try { *settings = new Settings(); COASTER_CHECK_MALLOC(*settings); @@ -129,7 +129,7 @@ coaster_rc coaster_settings_parse(coaster_settings *settings, const char *str, size_t str_len) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { // TODO: parsing using code currently in CoasterSwig return COASTER_ERROR_UNKNOWN; @@ -138,7 +138,7 @@ coaster_rc coaster_settings_set(coaster_settings *settings, const char *key, size_t key_len, - const char *value, size_t value_len) COASTERS_THROWS_NOTHING { + const char *value, size_t value_len) COASTER_THROWS_NOTHING { try { settings->set(key, key_len, value, value_len); return COASTER_SUCCESS; @@ -152,7 +152,7 @@ coaster_rc coaster_settings_get(coaster_settings *settings, const char *key, size_t key_len, - const char **value, size_t *value_len) COASTERS_THROWS_NOTHING { + const char **value, size_t *value_len) COASTER_THROWS_NOTHING { try { std::map &map = settings->getSettings(); std::string &str_value = map[string(key, key_len)]; @@ -169,7 +169,7 @@ coaster_rc coaster_settings_keys(coaster_settings *settings, const char ***keys, size_t **key_lens, int *count) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { try { std::map &map = settings->getSettings(); *count = map.size(); @@ -206,7 +206,7 @@ void coaster_settings_free(coaster_settings *settings) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { // Call destructor directly // Destructor shouldn't throw anything delete settings; @@ -218,7 +218,7 @@ coaster_rc coaster_apply_settings(coaster_client *client, coaster_settings *settings) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (settings == NULL || client == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } @@ -237,7 +237,7 @@ coaster_job_create(const char *executable, size_t executable_len, int argc, const char **argv, const size_t *arg_lens, const char *job_manager, size_t job_manager_len, - coaster_job **job) COASTERS_THROWS_NOTHING { + coaster_job **job) COASTER_THROWS_NOTHING { try { assert(executable != NULL); Job *j = new Job(string(executable, executable_len)); @@ -262,7 +262,7 @@ } coaster_rc -coaster_job_free(coaster_job *job) COASTERS_THROWS_NOTHING { +coaster_job_free(coaster_job *job) COASTER_THROWS_NOTHING { // Destructor shouldn't throw anything delete job; return COASTER_SUCCESS; @@ -270,7 +270,7 @@ coaster_rc coaster_job_to_string(const coaster_job *job, char **str, size_t *str_len) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL || str == NULL || str_len == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid argument"); } @@ -296,7 +296,7 @@ const char *stdin_loc, size_t stdin_loc_len, const char *stdout_loc, size_t stdout_loc_len, const char *stderr_loc, size_t stderr_loc_len) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } @@ -325,7 +325,7 @@ coaster_rc coaster_job_set_directory(coaster_job *job, const char *dir, size_t dir_len) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, ""); } @@ -346,7 +346,7 @@ coaster_rc coaster_job_set_envs(coaster_job *job, int nvars, const char **names, size_t *name_lens, - const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING { + const char **values, size_t *value_lens) COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid arg"); } @@ -378,7 +378,7 @@ coaster_rc coaster_job_set_attrs(coaster_job *job, int nattrs, const char **names, size_t *name_lens, - const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING { + const char **values, size_t *value_lens) COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); @@ -407,7 +407,7 @@ coaster_rc coaster_job_add_cleanups(coaster_job *job, int ncleanups, const char **cleanups, size_t *cleanup_lens) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); @@ -434,7 +434,7 @@ coaster_job_add_stages(coaster_job *job, int nstageins, coaster_stage_entry *stageins, int nstageouts, coaster_stage_entry *stageouts) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); @@ -463,14 +463,14 @@ } coaster_job_id -coaster_job_get_id(const coaster_job *job) COASTERS_THROWS_NOTHING { +coaster_job_get_id(const coaster_job *job) COASTER_THROWS_NOTHING { // Shouldn't throw anything from accessor method return job->getIdentity(); } coaster_rc coaster_job_status_code(const coaster_job *job, coaster_job_status *code) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { const JobStatus *status; if (job == NULL || (status = job->getStatus()) == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, @@ -485,7 +485,7 @@ coaster_job_get_outstreams(const coaster_job *job, const char **stdout_s, size_t *stdout_len, const char **stderr_s, size_t *stderr_len) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (job == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid job"); } @@ -514,7 +514,7 @@ coaster_rc coaster_submit(coaster_client *client, coaster_job *job) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { try { client->client.submit(*job); return COASTER_SUCCESS; @@ -528,7 +528,7 @@ coaster_rc coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, coaster_job **jobs, int *njobs) - COASTERS_THROWS_NOTHING { + COASTER_THROWS_NOTHING { if (client == NULL) { return coaster_return_error(COASTER_ERROR_INVALID, "invalid client"); } Index: modules/provider-coaster-c-client/src/coasters.h =================================================================== --- modules/provider-coaster-c-client/src/coasters.h (revision 3991) +++ modules/provider-coaster-c-client/src/coasters.h (working copy) @@ -89,9 +89,9 @@ // Set appropriate macro to specify that we shouldn't throw exceptions #ifdef __cplusplus -#define COASTERS_THROWS_NOTHING throw() +#define COASTER_THROWS_NOTHING throw() #else -#define COASTERS_THROWS_NOTHING +#define COASTER_THROWS_NOTHING #endif /* @@ -103,7 +103,7 @@ */ coaster_rc coaster_client_start(const char *service_url, size_t service_url_len, - coaster_client **client) COASTERS_THROWS_NOTHING; + coaster_client **client) COASTER_THROWS_NOTHING; /* * Stop coasters client and free all memory. @@ -112,7 +112,7 @@ * an argument to any more Coaster C API function calls. */ coaster_rc -coaster_client_stop(coaster_client *client) COASTERS_THROWS_NOTHING; +coaster_client_stop(coaster_client *client) COASTER_THROWS_NOTHING; /* * Create empty settings object @@ -121,14 +121,14 @@ */ coaster_rc coaster_settings_create(coaster_settings **settings) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Free memory associated with coasters settings */ void coaster_settings_free(coaster_settings *settings) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Parse settings from string. @@ -139,14 +139,14 @@ */ coaster_rc coaster_settings_parse(coaster_settings *settings, const char *str, - size_t str_len) COASTERS_THROWS_NOTHING; + size_t str_len) COASTER_THROWS_NOTHING; /* * Set settings individually. */ coaster_rc coaster_settings_set(coaster_settings *settings, const char *key, size_t key_len, - const char *value, size_t value_len) COASTERS_THROWS_NOTHING; + const char *value, size_t value_len) COASTER_THROWS_NOTHING; /* * Get settings individually. * value[_len]: set to value of string, null if not present in settings. @@ -156,7 +156,7 @@ coaster_rc coaster_settings_get(coaster_settings *settings, const char *key, size_t key_len, - const char **value, size_t *value_len) COASTERS_THROWS_NOTHING; + const char **value, size_t *value_len) COASTER_THROWS_NOTHING; /* * Enumerate settings. @@ -169,7 +169,7 @@ coaster_rc coaster_settings_keys(coaster_settings *settings, const char ***keys, size_t **key_lens, int *count) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Apply settings to started coasters client. @@ -179,7 +179,7 @@ coaster_rc coaster_apply_settings(coaster_client *client, coaster_settings *settings) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Create a new coasters job for later submission. @@ -195,13 +195,13 @@ coaster_job_create(const char *executable, size_t executable_len, int argc, const char **argv, const size_t *arg_lens, const char *job_manager, size_t job_manager_len, - coaster_job **job) COASTERS_THROWS_NOTHING; + coaster_job **job) COASTER_THROWS_NOTHING; /* * Free a coasters job */ coaster_rc -coaster_job_free(coaster_job *job) COASTERS_THROWS_NOTHING; +coaster_job_free(coaster_job *job) COASTER_THROWS_NOTHING; /* * Create a human readable string describing job. @@ -209,7 +209,7 @@ */ coaster_rc coaster_job_to_string(const coaster_job *job, char **str, size_t *str_len) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Set input and output stream redirections. @@ -220,7 +220,7 @@ const char *stdin_loc, size_t stdin_loc_len, const char *stdout_loc, size_t stdout_loc_len, const char *stderr_loc, size_t stderr_loc_len) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Set job directory. @@ -228,7 +228,7 @@ */ coaster_rc coaster_job_set_directory(coaster_job *job, const char *dir, size_t dir_len) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Add environment variables for the job. Will overwrite any @@ -238,7 +238,7 @@ coaster_rc coaster_job_set_envs(coaster_job *job, int nvars, const char **names, size_t *name_lens, - const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING; + const char **values, size_t *value_lens) COASTER_THROWS_NOTHING; /* * Add attributes for the job. Will overwrite any previous atrributes @@ -248,7 +248,7 @@ coaster_rc coaster_job_set_attrs(coaster_job *job, int nattrs, const char **names, size_t *name_lens, - const char **values, size_t *value_lens) COASTERS_THROWS_NOTHING; + const char **values, size_t *value_lens) COASTER_THROWS_NOTHING; /* * Add cleanups for job. @@ -256,7 +256,7 @@ coaster_rc coaster_job_add_cleanups(coaster_job *job, int ncleanups, const char **cleanups, size_t *cleanup_lens) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Add staging set entries for job @@ -265,14 +265,14 @@ coaster_job_add_stages(coaster_job *job, int nstageins, coaster_stage_entry *stageins, int nstageouts, coaster_stage_entry *stageouts) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Get local job ID. The job ID is a locally unique identifier for * a coasters job that is assigned when the job is created. */ coaster_job_id -coaster_job_get_id(const coaster_job *job) COASTERS_THROWS_NOTHING; +coaster_job_get_id(const coaster_job *job) COASTER_THROWS_NOTHING; /* * Get status of a submitted job. @@ -280,7 +280,7 @@ */ coaster_rc coaster_job_status_code(const coaster_job *job, coaster_job_status *code) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Get stdin/out of completed job. @@ -291,7 +291,7 @@ coaster_job_get_outstreams(const coaster_job *job, const char **stdout_s, size_t *stdout_len, const char **stderr_s, size_t *stderr_len) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Submit a coasters job through a coasters client. @@ -302,7 +302,7 @@ */ coaster_rc coaster_submit(coaster_client *client, coaster_job *job) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Check for completion of jobs. @@ -315,7 +315,7 @@ coaster_rc coaster_check_jobs(coaster_client *client, bool wait, int maxjobs, coaster_job **jobs, int *njobs) - COASTERS_THROWS_NOTHING; + COASTER_THROWS_NOTHING; /* * Get name of return code. Returns NULL if invalid code. From hategan at ci.uchicago.edu Sat Jun 28 13:49:52 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 28 Jun 2014 13:49:52 -0500 (CDT) Subject: [Swift-commit] r7930 - trunk/src/org/griphyn/vdl/mapping/file Message-ID: <20140628184952.CC393178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-28 13:49:51 -0500 (Sat, 28 Jun 2014) New Revision: 7930 Modified: trunk/src/org/griphyn/vdl/mapping/file/ConcurrentElementMapper.java Log: made index mapping for the concurrent mapper more sane: removed funny things like "array" and "elt"; use a naming structure based on the actual indices that is human readable Modified: trunk/src/org/griphyn/vdl/mapping/file/ConcurrentElementMapper.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/file/ConcurrentElementMapper.java 2014-06-24 21:31:30 UTC (rev 7929) +++ trunk/src/org/griphyn/vdl/mapping/file/ConcurrentElementMapper.java 2014-06-28 18:49:51 UTC (rev 7930) @@ -28,15 +28,8 @@ public class ConcurrentElementMapper extends AbstractFileNameElementMapper { - /** determines how many directories and element files are permitted - in each directory. There will be no more than - DIRECTORY_LOAD_FACTOR element files and no more than - DIRECTORY_LOAD_FACTOR directories, so there could be up to - 2 * DIRECTORY_LOAD_FACTOR elements. */ - public final static int DIRECTORY_LOAD_FACTOR=25; - public String mapField(String fieldName) { - return "-field/"+fieldName; + return fieldName; } public String rmapField(String pathElement) { @@ -44,33 +37,96 @@ } public String mapIndex(int index, int pos) { - StringBuffer sb = new StringBuffer(); - sb.append("-array/"); - sb.append(splitIndexByLoadFactor(index)); - sb.append("/"); - sb.append("elt-"); - sb.append(String.valueOf(index)); - return sb.toString(); + return indexToPath(index); } - - String splitIndexByLoadFactor(int index) { - if (index <= DIRECTORY_LOAD_FACTOR) { - return ""; - } - else { - String prefix = "h" + String.valueOf(index % DIRECTORY_LOAD_FACTOR) + "/"; - String suffix = splitIndexByLoadFactor(index / DIRECTORY_LOAD_FACTOR); - return prefix + suffix; - } + /** + * Indices 0 to 9 -> [0] ... [9] + * Indices 10 to 99 -> [d0-d9]/[d0]...[d9] + * Indices 100 to 999 -> [d00-d99]/[de0-de9]/... + * ... + * etc. + */ + public static String indexToPath(int index) { + boolean neg = false; + if (index < 0) { + neg = true; + index = -index; + } + + StringBuilder sb = new StringBuilder(); + sb.append("/"); + indexToPath(sb, neg, 0, index); + return sb.toString(); } + public static void indexToPath(StringBuilder sb, boolean negative, int prefix, int index) { + if (index < 10) { + sb.append('['); + if (negative) { + sb.append('-'); + } + if (prefix != 0) { + sb.append(prefix / 10); + } + sb.append(index); + sb.append(']'); + } + else { + int t = getFirstDigit(index); + int o = getOrder(index); + sb.append("["); + if (negative) { + sb.append('-'); + } + if (prefix != 0) { + sb.append(prefix / o / 10); + } + sb.append(t); + sb.append('-'); + if (prefix != 0) { + sb.append(prefix / o / 10); + } + sb.append(t + o - o / 10); + sb.append("]/"); + indexToPath(sb, negative, prefix + t, index - t); + } + } + + /** + * Assume n >= 10 + * Return n with all but the most significant digit + * set to 0 + */ + private static int getFirstDigit(int n) { + int order = 1; + while (n >= 10) { + n = n / 10; + order = order * 10; + } + return n * order; + } + + /** + * Assume n >= 10 + * Return 1 followed by the number of digits in n minus one + */ + private static int getOrder(int n) { + int order = 1; + while (n >= 10) { + n = n / 10; + order = order * 10; + } + return order; + } + + // TODO: this should be but is not the inverse of mapIndex public int rmapIndex(String pathElement) { return Integer.parseInt(pathElement); } public String getSeparator(int depth) { return "_"; - } + } } From hategan at ci.uchicago.edu Sat Jun 28 13:50:25 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 28 Jun 2014 13:50:25 -0500 (CDT) Subject: [Swift-commit] r7931 - trunk/src/org/griphyn/vdl/mapping/file Message-ID: <20140628185025.DF821178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-28 13:50:25 -0500 (Sat, 28 Jun 2014) New Revision: 7931 Modified: trunk/src/org/griphyn/vdl/mapping/file/ExternalMapper.java Log: removed wasteful debugging line Modified: trunk/src/org/griphyn/vdl/mapping/file/ExternalMapper.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/file/ExternalMapper.java 2014-06-28 18:49:51 UTC (rev 7930) +++ trunk/src/org/griphyn/vdl/mapping/file/ExternalMapper.java 2014-06-28 18:50:25 UTC (rev 7931) @@ -83,14 +83,10 @@ } Process p = Runtime.getRuntime().exec(cmd.toArray(STRING_ARRAY)); process(p.getInputStream()); - List lines = fetchOutput(p.getInputStream()); - if (logger.isDebugEnabled()) { - logger.debug("external mapper output: " + lines); - } int ec = p.waitFor(); if (ec != 0) { throw new RuntimeException("External executable failed. Exit code: " + ec + "\n\t" - + join(lines) + "\n\t" + join(fetchOutput(p.getErrorStream()))); + + join(fetchOutput(p.getErrorStream()))); } } catch (IOException e) { From hategan at ci.uchicago.edu Sat Jun 28 13:50:40 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 28 Jun 2014 13:50:40 -0500 (CDT) Subject: [Swift-commit] r7932 - trunk/src/org/griphyn/vdl/mapping/file Message-ID: <20140628185040.431F7178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-28 13:50:40 -0500 (Sat, 28 Jun 2014) New Revision: 7932 Modified: trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java Log: cleaned up a bit Modified: trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java =================================================================== --- trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java 2014-06-28 18:50:25 UTC (rev 7931) +++ trunk/src/org/griphyn/vdl/mapping/file/SingleFileMapper.java 2014-06-28 18:50:40 UTC (rev 7932) @@ -31,6 +31,7 @@ /** Maps every Path to a single file name (specified by the "file" parameter). */ public class SingleFileMapper extends AbstractMapper { + public static final String NAME = "SingleFileMapper"; @Override protected void getValidMappingParams(Set s) { @@ -49,7 +50,7 @@ @Override public String getName() { - return "SingleFileMapper"; + return NAME; } @Override From hategan at ci.uchicago.edu Sat Jun 28 13:52:47 2014 From: hategan at ci.uchicago.edu (hategan at ci.uchicago.edu) Date: Sat, 28 Jun 2014 13:52:47 -0500 (CDT) Subject: [Swift-commit] r7933 - trunk/src/org/griphyn/vdl/engine Message-ID: <20140628185247.03276178884@svn.ci.uchicago.edu> Author: hategan Date: 2014-06-28 13:52:46 -0500 (Sat, 28 Jun 2014) New Revision: 7933 Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java Log: some cleanups and removed UUID from concurrent mapper since each run gets its own directory now, so it's just useless noise Modified: trunk/src/org/griphyn/vdl/engine/Karajan.java =================================================================== --- trunk/src/org/griphyn/vdl/engine/Karajan.java 2014-06-28 18:50:40 UTC (rev 7932) +++ trunk/src/org/griphyn/vdl/engine/Karajan.java 2014-06-28 18:52:46 UTC (rev 7933) @@ -549,8 +549,7 @@ mappingST.setAttribute("descriptor", "ConcurrentMapper"); StringTemplate paramST = template("swift_parameter"); paramST.setAttribute("name", "prefix"); - paramST.setAttribute("expr", "\"" + var.getName() + "-" - + UUIDGenerator.getInstance().generateRandomBasedUUID().toString() + "\""); + paramST.setAttribute("expr", "\"" + var.getName() + "-" + lineNumber(var.getSrc()) + "\""); mappingST.setAttribute("params", paramST); variableST.setAttribute("mapping", mappingST); variableST.setAttribute("nil", Boolean.TRUE); @@ -1142,7 +1141,7 @@ StringTemplate foreachST = template("foreach"); foreachST.setAttribute("var", foreach.getVar()); - foreachST.setAttribute("line", foreach.getSrc().substring(foreach.getSrc().indexOf(' ') + 1)); + foreachST.setAttribute("line", lineNumber(foreach.getSrc())); XmlObject in = foreach.getIn().getAbstractExpression(); StringTemplate inST = expressionToKarajan(in, scope); @@ -2052,10 +2051,14 @@ private int fieldCounter = 1; private String addInternedField(String name, String type) { - String v = "swift.field." + name + "." + type.replace("[", ".array.").replace("]", ""); + String v = internedFieldName(name, type); usedFields.put(new InternedField(name, type), v); return v; } + + public static String internedFieldName(String name, String type) { + return "swift.field." + name + "." + type.replace("[", ".array.").replace("]", ""); + } public void generateInternedConstants(StringTemplate programTemplate) { generateInternedConstants(programTemplate, stringInternMap, "sConst"); @@ -2107,6 +2110,10 @@ public static String normalize(String type) { return org.griphyn.vdl.type.Types.normalize(type); } + + public static String lineNumber(String src) { + return src.substring(src.indexOf(' ') + 1); + } String datatype(StringTemplate st) { try { From swift at ci.uchicago.edu Mon Jun 30 12:15:03 2014 From: swift at ci.uchicago.edu (swift at ci.uchicago.edu) Date: Mon, 30 Jun 2014 12:15:03 -0500 (CDT) Subject: [Swift-commit] cog r3993 Message-ID: <20140630171503.9BCB08D000A9@bridled.ci.uchicago.edu> ------------------------------------------------------------------------ r3993 | timgarmstrong | 2014-06-30 12:10:19 -0500 (Mon, 30 Jun 2014) | 1 line Static linkage for internal functions ------------------------------------------------------------------------ Index: modules/provider-coaster-c-client/src/CoasterChannel.cpp =================================================================== --- modules/provider-coaster-c-client/src/CoasterChannel.cpp (revision 3992) +++ modules/provider-coaster-c-client/src/CoasterChannel.cpp (working copy) @@ -27,8 +27,8 @@ using std::min; using std::string; -int socksend(int fd, const char* buf, int len, int flags); -void pp(char* dest, const char* src, int len); +static int socksend(int fd, const char* buf, int len, int flags); +static void pp(char* dest, const char* src, int len); class HeartBeatCommand; @@ -357,11 +357,11 @@ /* * Without this, GCC gets confused by CoasterChannel::send. */ -int socksend(int fd, const char* buf, int len, int flags) { +static int socksend(int fd, const char* buf, int len, int flags) { return send(fd, buf, len, flags); } -void pp(char* dest, const char* src, int len) { +static void pp(char* dest, const char* src, int len) { for(int i = 0; i < len; i++) { char c = src[i]; if (c < '0' || c > 127) { From yadunandb at ci.uchicago.edu Mon Jun 9 17:36:21 2014 From: yadunandb at ci.uchicago.edu (yadunandb at ci.uchicago.edu) Date: Mon, 09 Jun 2014 22:36:21 -0000 Subject: [Swift-commit] r7910 - in SwiftTutorials/swift-localhost-tutorial: . app bin doc doc/figs doc/images doc/images/icons part01 part01/run001 part01/run002 part01/run003 part01/run004 part01/run005 part02 part03 part04 Message-ID: <20140609223736.B51E49D053@svn.ci.uchicago.edu> Author: yadunandb Date: 2014-06-09 17:37:36 -0500 (Mon, 09 Jun 2014) New Revision: 7910 Added: SwiftTutorials/swift-localhost-tutorial/README.asc SwiftTutorials/swift-localhost-tutorial/app/ SwiftTutorials/swift-localhost-tutorial/app/simulate SwiftTutorials/swift-localhost-tutorial/app/simulate.py SwiftTutorials/swift-localhost-tutorial/app/simulate.sh SwiftTutorials/swift-localhost-tutorial/app/stats SwiftTutorials/swift-localhost-tutorial/app/stats.py SwiftTutorials/swift-localhost-tutorial/app/stats.sh SwiftTutorials/swift-localhost-tutorial/bin/ SwiftTutorials/swift-localhost-tutorial/bin/cleanup SwiftTutorials/swift-localhost-tutorial/bin/hosts SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh SwiftTutorials/swift-localhost-tutorial/bin/plot.sh SwiftTutorials/swift-localhost-tutorial/doc/ SwiftTutorials/swift-localhost-tutorial/doc/README SwiftTutorials/swift-localhost-tutorial/doc/TODO SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png SwiftTutorials/swift-localhost-tutorial/doc/figs/ SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png SwiftTutorials/swift-localhost-tutorial/doc/images/ SwiftTutorials/swift-localhost-tutorial/doc/images/icons/ SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png SwiftTutorials/swift-localhost-tutorial/doc/part01.png SwiftTutorials/swift-localhost-tutorial/doc/part02.png SwiftTutorials/swift-localhost-tutorial/doc/part03.png SwiftTutorials/swift-localhost-tutorial/doc/part04.png SwiftTutorials/swift-localhost-tutorial/doc/part05.png SwiftTutorials/swift-localhost-tutorial/doc/part06.png SwiftTutorials/swift-localhost-tutorial/doc/push.sh SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html SwiftTutorials/swift-localhost-tutorial/part01/ SwiftTutorials/swift-localhost-tutorial/part01/p1.swift SwiftTutorials/swift-localhost-tutorial/part01/run001/ SwiftTutorials/swift-localhost-tutorial/part01/run001/apps SwiftTutorials/swift-localhost-tutorial/part01/run001/cf SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out SwiftTutorials/swift-localhost-tutorial/part01/run002/ SwiftTutorials/swift-localhost-tutorial/part01/run002/apps SwiftTutorials/swift-localhost-tutorial/part01/run002/cf SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out SwiftTutorials/swift-localhost-tutorial/part01/run003/ SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out SwiftTutorials/swift-localhost-tutorial/part01/run004/ SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out SwiftTutorials/swift-localhost-tutorial/part01/run005/ SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out SwiftTutorials/swift-localhost-tutorial/part01/sim.out SwiftTutorials/swift-localhost-tutorial/part01/swift.properties SwiftTutorials/swift-localhost-tutorial/part02/ SwiftTutorials/swift-localhost-tutorial/part02/p2.swift SwiftTutorials/swift-localhost-tutorial/part02/swift.properties SwiftTutorials/swift-localhost-tutorial/part03/ SwiftTutorials/swift-localhost-tutorial/part03/p3.swift SwiftTutorials/swift-localhost-tutorial/part03/swift.properties SwiftTutorials/swift-localhost-tutorial/part04/ SwiftTutorials/swift-localhost-tutorial/part04/p4.swift SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh SwiftTutorials/swift-localhost-tutorial/part04/stats.sh SwiftTutorials/swift-localhost-tutorial/part04/swift.properties SwiftTutorials/swift-localhost-tutorial/setup.sh SwiftTutorials/swift-localhost-tutorial/test_all.sh Log: Committing localhost tutorial. Live copy on git: https://github.com/yadudoc/swift-on-cloud/tree/master/swift-localhost-tutorial Added: SwiftTutorials/swift-localhost-tutorial/README.asc =================================================================== --- SwiftTutorials/swift-localhost-tutorial/README.asc (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/README.asc 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +link doc/README \ No newline at end of file Property changes on: SwiftTutorials/swift-localhost-tutorial/README.asc ___________________________________________________________________ Added: svn:special + * Added: SwiftTutorials/swift-localhost-tutorial/app/simulate =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/simulate (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/simulate 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +link simulate.sh \ No newline at end of file Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate ___________________________________________________________________ Added: svn:special + * Added: SwiftTutorials/swift-localhost-tutorial/app/simulate.py =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/simulate.py (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/simulate.py 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,125 @@ +#!/usr/bin/env python +import sys +import os +import socket +import time +import math +import random +import getpass + +def parse(): + from optparse import OptionParser + parser=OptionParser() + parser.add_option("-b", "--bias", type="int", dest="bias", default=0, + help="offset bias: add this integer to all results"); + + parser.add_option("-B", "--biasfile", type="string", dest="biasfile", + default="none", help="file of integer biases to add to results" ); + + parser.add_option("-l", "--log", type="string", dest="log", default="yes", + help="generate a log in stderr if not nul"); + + parser.add_option("-n", "--nvalues", type="int", dest="nvalues",default=1, + help="print this many values per simulation" ); + + parser.add_option("-s", "--seed", type="int", dest="initseed", default=0, + help="use this integer [0..32767] as a seed"); + + parser.add_option("-S", "--seedfile", type="string", dest="seedfile", + default="none", help="use this file (containing integer seeds [0..32767]) one per line" ); + + parser.add_option("-t", "--timesteps", type="int", dest="timesteps", + default=0, help='number of simulated "timesteps" in seconds (determines runtime)' ); + + parser.add_option("-r", "--range", type="int", dest="range", default=100, + help="range (limit) of generated results" ); + + parser.add_option("-w", "--width", type="int", dest="width", default=8, + help="Width ?" ); + + parser.add_option("-x", "--scale", type="int", dest="scale", default=1, + help="scale the results by this integer" ); + # Not implemented yet + parser.add_option("-p", "--paramfile", type="string", dest="paramfile", default="none", + help="Not implemented yet" ); + return (parser.parse_args()); + +def log(): + import datetime + print >> sys.stderr, "Called as: ", str(sys.argv) + print >> sys.stderr, "Start time: ", datetime.datetime.now() + print >> sys.stderr, "Running on node: ", socket.gethostname() + print >> sys.stderr, "Running as user: ", getpass.getuser() + print >> sys.stderr, "\nEnvironment:\n\n" + print >> sys.stderr, os.environ + +def printparams(options): + print 'bias =',options.bias + print 'biasfile = ',options.biasfile + print 'log = ', options.log + print 'nvalues = ',options.nvalues + print 'seed = ', options.initseed + print 'seedfile = ', options.seedfile + print 'timesteps = ', options.timesteps + print 'range = ', options.range + print 'width = ', options.width + print 'scale = ', options.scale + print 'paramfile = ', options.paramfile + +def simulate(options): + time.sleep(options.timesteps) + bias=[]; + if (options.biasfile != "none"): + try: + with open(options.biasfile) as biasfile: + lines = biasfile.read().splitlines() + for line in lines: + bias.append(int(line)) + except IOError: + print "Error accessing content from file: ", options.biasfile + bias_count = len(bias) + + for i in range(options.nvalues): + value = (random.random() + + random.random()*math.pow(2,16) + + random.random()*math.pow(2,32) + + random.random()*math.pow(2,48)) + value=( (int(value)%options.range) * options.scale + options.bias) + if ( i < bias_count ): + value = value + bias[i] + elif ( bias_count > 0 ): + value = value + bias[bias_count-1] + + print '{num:{fill}{width}}'.format(num=value, fill=' ', width=options.width) + + +def seed(options): + if (options.initseed != 0 ): + random.seed(options.initseed) + if (options.seedfile != "none"): + try: + with open(options.seedfile) as seedfile: + lines=seedfile.read().splitlines() + seed = 0 + for line in lines: + seed = seed + int(line) + random.seed(seed) + except IOError: + print "Could not open file: ", options.seedfile + + +if __name__ == "__main__": + (options, args) = parse() + printparams(options) + seed(options) + simulate(options) + log() + + + + + + + + + Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate.py ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/app/simulate.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/simulate.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/simulate.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,135 @@ +#! /bin/bash + +printparams() +{ + printf "\nSimulation parameters:\n\n" + echo bias=$bias + echo biasfile=$biasfile + echo initseed=$initseed + echo log=$log + echo paramfile=$paramfile + echo range=$range + echo scale=$scale + echo seedfile=$seedfile + echo timesteps=$timesteps + echo output width=$width +} + +log() { + printf "\nCalled as: $0: $cmdargs\n\n" + printf "Start time: "; /bin/date + printf "Running as user: "; /usr/bin/id + printf "Running on node: "; /bin/hostname + printf "Node IP address: "; /bin/hostname -I + printparams + printf "\nEnvironment:\n\n" + printenv | sort +} + +addsims() { + while read f1 ; do + read -u 3 f2 + if [ _$f1 = _ ]; then f1=$lastf1; fi + if [ _$f2 = _ ]; then f2=$lastf2; fi + printf "%${width}d\n" $(($f1+$f2)) + lastf1=$f1 + lastf2=$f2 + done <$1 3<$2 +} + +# set defaults + +bias=0 +biasfile=none +initseed=none +log=yes +paramfile=none +range=100 +scale=1 +seedfile=none +timesteps=0 +nvalues=1 +width=8 +cmdargs="$*" + +usage() +{ + echo $0: usage: + cat <$simout + +# process file of biases + +if [ $biasfile != none ]; then + addsims $simout $biasfile +else + cat $simout +fi +rm $simout + +# log environmental data + +if [ $log != off ]; then + log 1>&2 +fi Property changes on: SwiftTutorials/swift-localhost-tutorial/app/simulate.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/app/stats =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/stats (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/stats 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +link stats.sh \ No newline at end of file Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats ___________________________________________________________________ Added: svn:special + * Added: SwiftTutorials/swift-localhost-tutorial/app/stats.py =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/stats.py (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/stats.py 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,31 @@ +#!/usr/bin/env python +import socket +import sys +import os + +def log(): + import datetime + print >> sys.stderr, "Called as: ", str(sys.argv) + print >> sys.stderr, "Start time: ", datetime.datetime.now() + print >> sys.stderr, "Running on node: ", socket.gethostname() + print >> sys.stderr, "Running as user: ",os.getlogin() + print >> sys.stderr, "\nEnvironment:\n\n" + print >> sys.stderr, os.environ + +def stat(): + total = 0 + count = 0 + for f in sys.argv[1:]: + try: + with open(f) as ifile: + lines = ifile.read().splitlines() + for line in lines: + total += int(line) + count += 1 + except IOError: + print "Error accessing content from file: ", options.biasfile + print total/count + +if __name__ == "__main__": + log() + stat() Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats.py ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/app/stats.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/app/stats.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/app/stats.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,19 @@ +#! /bin/sh + +log() { + printf "\nCalled as: $0: $cmdargs\n\n" + printf "Start time: "; /bin/date + printf "Running as user: "; /usr/bin/id + printf "Running on node: "; /bin/hostname + printf "Node IP address: "; /bin/hostname -I + printf "\nEnvironment:\n\n" + printenv | sort +} + +awk ' + +{ sum += $1} + +END { printf("%d\n",sum/NR) } +' $* +log 1>&2 Property changes on: SwiftTutorials/swift-localhost-tutorial/app/stats.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/bin/cleanup =================================================================== --- SwiftTutorials/swift-localhost-tutorial/bin/cleanup (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/bin/cleanup 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,2 @@ +rm -rf *.log *.rlog *.d p?-*-*-* *.kml *.swiftx *.out output outdir logs hi.* _concurrent .swift/tmp run??? + Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/cleanup ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/bin/hosts =================================================================== --- SwiftTutorials/swift-localhost-tutorial/bin/hosts (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/bin/hosts 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +for h in $(cat hosts.txt); do ssh $h hostname -f; done Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/hosts ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,15 @@ +#! /bin/sh + +BIN=$(cd $(dirname $0); pwd) +BASE=$(cd $BIN/..; pwd) +LOC=$(cd $BASE/..; pwd) + +echo executing $0 from BIN=$BIN +echo making tutorial package of BASE=$BASE +echo placing tutorial package in LOC=$LOC + +( + cd $LOC + tar zcf swift-cray-tutorial.tgz --exclude-vcs swift-cray-tutorial + scp swift-cray-tutorial.tgz p01532 at raven.cray.com: +) Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/package_tutorial.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/bin/plot.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/bin/plot.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/bin/plot.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,68 @@ +#!/bin/bash + +#usage: ./plotswiftlogs.ketan + +SWIFTLOGFILE=$1 + +#TMPDIR=`mktemp -d plotlog.XXX` + +grep -i ProgressTicker $SWIFTLOGFILE > swiftoutfile.out + +SWIFTOUTFILE=swiftoutfile.out + +#extract start time +TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | head -n 1 | cut -f1-2 -d ' '` +START_TIME=`date +%s -d "$TMPDATE"` + +#extract end time +TMPDATE=`grep -i progress $SWIFTOUTFILE 2>/dev/null | tail -n 1 | cut -f1-2 -d ' '` +END_TIME=`date +%s -d "$TMPDATE"` + +#duration +DIFFTIME=$((END_TIME - START_TIME)) + +#extract active runs in a file +(grep -o -i "Active:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' >active.txt) + +#extract successful completions in a file +(grep -o -i "Successfully:[0-9]*" $SWIFTOUTFILE 2>/dev/null | awk -F: '{print $2}' > cumulative.txt) + +#prepare tics +activelines=`wc -l active.txt | awk '{print $1}'` +cumulines=`wc -l cumulative.txt | awk '{print $1}'` + +if [ $activelines -ne 0 ] +then + activelinespertic=`echo "scale=5 ; $DIFFTIME / $activelines" | bc` +fi + +seq 0 $activelinespertic $DIFFTIME > activetics.txt + +if [ $cumulines -ne 0 ] +then + cumulinespertic=`echo "scale=5 ; $DIFFTIME / $cumulines" | bc` +fi + +seq 0 $cumulinespertic $DIFFTIME > cumultics.txt + +#final plot data +paste activetics.txt active.txt > plot_active.txt +paste cumultics.txt cumulative.txt > plot_cumulative.txt + +cat << EOF1 > plotit.gp +set terminal png enhanced +set nokey +set output "cumulativeplot.png" +set xlabel "Time in sec" +set ylabel "number of completed apps" +set title "Cumulative Apps Completed" +plot "plot_cumulative.txt" using 1:2 with lines +set output "activeplot.png" +set xlabel "Time in sec" +set ylabel "number of active apps" +set title "Active Apps" +plot "plot_active.txt" using 1:2 with line +EOF1 + +gnuplot plotit.gp 2>/dev/null +rm swiftoutfile.out plotit.gp active.txt cumulative.txt activetics.txt plot_active.txt plot_cumulative.txt cumultics.txt Property changes on: SwiftTutorials/swift-localhost-tutorial/bin/plot.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/doc/README =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/README (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/README 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,619 @@ +Swift Tutorial for running on localhost +======================================= + +This tutorial is viewable at: +http://swift-lang.org/tutorials/localhost/tutorial.html + + +Introduction: Why Parallel Scripting? +------------------------------------ + +Swift is a simple scripting language for executing many instances of +ordinary application programs on distributed parallel resources. +Swift scripts run many copies of ordinary programs concurrently, using +statements like this: +----- +file out[]; +foreach param,i in paramList { + out[i] = simulate(param); +} +----- + +Swift acts like a structured "shell" language. It runs programs +concurrently as soon as their inputs are available, reducing the need +for complex parallel programming. Swift expresses your workflow in a +portable fashion: The same script runs on multicore computers, +clusters, clouds, grids, and supercomputers. + +In this tutorial, you'll be able to first try a few Swift examples +on your local machine, to get a sense of the language. + +//// +Then in parts 4-6 you'll run similar workflows on cloud +compute nodes, and see how more complex workflows can be expressed +with Swift scripts. +//// + +Swift installation +------------------ +1. Download the file from http://swiftlang.org/packages/swift-0.95-RC6.tar.gz +2. Extract by running "tar xfz swift-0.95-RC6.tar.gz" +3. Add to PATH by running "export PATH=$PATH:/path/to/swift-0.95-RC6/bin" + +Swift tutorial installation +--------------------------- +Run the following commands to extract these tutorial scripts. + +----- +$ cd $HOME +$ wget http://swiftlang.org/tutorials/localhost/swift-localhost-tutorial.tar.gz +$ tar xvfz swift-localhost-tutorial.tar.gz +$ cd swift-localhost-tutorial +----- + +Run the tutorial setup script +----- +$ source setup.sh # You must run this with "source" ! +----- + +To check out the tutorial scripts from SVN +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you later want to get the most recent version of this tutorial from +the Swift Subversion repository, do: + +----- +$ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/swift-localhost-tutorial +----- + +This will create a directory called "swift-localhost-tutorial" which contains all of the +files used in this tutorial. + + +Simple "science applications" for the workflow tutorial +------------------------------------------------------- + +This tutorial is based on two intentionally trivial example programs, +`simulation.sh` and `stats.sh`, (implemented as bash shell scripts) +that serve as easy-to-understand proxies for real science +applications. These "programs" behave as follows. + +simulate.sh +~~~~~~~~~~~ + +The simulation.sh script serves as a trivial proxy for any more +complex scientific simulation application. It generates and prints a +set of one or more random integers in the range [0-2^62) as controlled +by its command line arguments, which are: + +----- +$ ./app/simulate.sh --help +./app/simulate.sh: usage: + -b|--bias offset bias: add this integer to all results [0] + -B|--biasfile file of integer biases to add to results [none] + -l|--log generate a log in stderr if not null [y] + -n|--nvalues print this many values per simulation [1] + -r|--range range (limit) of generated results [100] + -s|--seed use this integer [0..32767] as a seed [none] + -S|--seedfile use this file (containing integer seeds [0..32767]) one per line [none] + -t|--timesteps number of simulated "timesteps" in seconds (determines runtime) [1] + -x|--scale scale the results by this integer [1] + -h|-?|?|--help print this help +$ +----- + +All of thess arguments are optional, with default values indicated above as `[n]`. + +//// +.simulation.sh arguments +[width="80%",cols="^2,10",options="header"] + +|======================= +|Argument|Short|Description +|1 |runtime: sets run time of simulation.sh in seconds +|2 |range: limits generated values to the range [0,range-1] +|3 |biasfile: add the integer contained in this file to each value generated +|4 |scale: multiplies each generated value by this integer +|5 |count: number of values to generate in the simulation +|======================= +//// + +With no arguments, simulate.sh prints 1 number in the range of +1-100. Otherwise it generates n numbers of the form (R*scale)+bias +where R is a random integer. By default it logs information about its +execution environment to stderr. Here's some examples of its usage: + +----- +$ simulate.sh 2>log + 5 +$ head -4 log + +Called as: /home/wilde/swift/tut/CIC_2013-08-09/app/simulate.sh: +Start time: Thu Aug 22 12:40:24 CDT 2013 +Running on node: login01.osgconnect.net + +$ simulate.sh -n 4 -r 1000000 2>log + 239454 + 386702 + 13849 + 873526 + +$ simulate.sh -n 3 -r 1000000 -x 100 2>log + 6643700 +62182300 + 5230600 + +$ simulate.sh -n 2 -r 1000 -x 1000 2>log + 565000 + 636000 + +$ time simulate.sh -n 2 -r 1000 -x 1000 -t 3 2>log + 336000 + 320000 +real 0m3.012s +user 0m0.005s +sys 0m0.006s +----- + +stats.sh +~~~~~~~~ + +The stats.sh script serves as a trivial model of an "analysis" +program. It reads N files each containing M integers and simply prints +the\ average of all those numbers to stdout. Similarly to simulate.sh +it logs environmental information to the stderr. + +----- +$ ls f* +f1 f2 f3 f4 + +$ cat f* +25 +60 +40 +75 + +$ stats.sh f* 2>log +50 +----- + + +Basics of the Swift language with local execution +------------------------------------------------- + +A Summary of Swift in a nutshell +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Swift scripts are text files ending in `.swift` The `swift` command +runs on any host, and executes these scripts. `swift` is a Java +application, which you can install almost anywhere. On Linux, just +unpack the distribution `tar` file and add its `bin/` directory to +your `PATH`. + +* Swift scripts run ordinary applications, just like shell scripts +do. Swift makes it easy to run these applications on parallel and +remote computers (from laptops to supercomputers). If you can `ssh` to +the system, Swift can likely run applications there. + +* The details of where to run applications and how to get files back +and forth are described in configuration files separate from your +program. Swift speaks ssh, PBS, Condor, SLURM, LSF, SGE, Cobalt, and +Globus to run applications, and scp, http, ftp, and GridFTP to move +data. + +* The Swift language has 5 main data types: `boolean`, `int`, +`string`, `float`, and `file`. Collections of these are dynamic, +sparse arrays of arbitrary dimension and structures of scalars and/or +arrays defined by the `type` declaration. + +* Swift file variables are "mapped" to external files. Swift sends +files to and from remote systems for you automatically. + +* Swift variables are "single assignment": once you set them you can't +change them (in a given block of code). This makes Swift a natural, +"parallel data flow" language. This programming model keeps your +workflow scripts simple and easy to write and understand. + +* Swift lets you define functions to "wrap" application programs, and +to cleanly structure more complex scripts. Swift `app` functions take +files and parameters as inputs and return files as outputs. + +* A compact set of built-in functions for string and file +manipulation, type conversions, high level IO, etc. is provided. +Swift's equivalent of `printf()` is `tracef()`, with limited and +slightly different format codes. + +* Swift's `foreach {}` statement is the main parallel workhorse of the +language, and executes all iterations of the loop concurrently. The +actual number of parallel tasks executed is based on available +resources and settable "throttles". + +* In fact, Swift conceptually executes *all* the statements, +expressions and function calls in your program in parallel, based on +data flow. These are similarly throttled based on available resources +and settings. + +* Swift also has `if` and `switch` statements for conditional +execution. These are seldom needed in simple workflows but they enable +very dynamic workflow patterns to be specified. + +We'll see many of these points in action in the examples below. Lets +get started! + +Part 1: Run a single application under Swift +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The first swift script, p1.swift, runs simulate.sh to generate a +single random number. It writes the number to a file. + +image::part01.png["p1 workflow",align="center"] + +.p1.swift +----- +sys::[cat ../part01/p1.swift] +----- + + +To run this script, run the following command: +----- +$ cd part01 +$ swift p1.swift +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run001 +Progress: Tue, 03 Jun 2014 15:01:28-0500 +Final status:Tue, 03 Jun 2014 15:01:29-0500 Finished successfully:1 +$ cat sim.out + 84 +$ swift p1.swift +$ cat sim.out + 36 +----- + +To cleanup the directory and remove all outputs (including the log +files and directories that Swift generates), run the cleanup script +which is located in the tutorial PATH: + +----- +$ cleanup +------ + +NOTE: You'll also find a Swift configuration files in each `partNN` +directory of this tutorial. These specify the environment-specific +details of where to find application programs and where to run them . +These files will be explained in more detail in parts 4-6, and can +be ignored for now. + +//// +It defines +things like the work directory, the scheduler to use, and how to +control parallelism. The sites.xml file below will tell Swift to run +on the local machine only, and run just 1 task at a time. + +.swift.properties +----- +sys::[cat ../part01/swift.properties] +----- + + In this case, it +indicates that the app "simulate" (the first token in the command line +declaration of the function `simulation`, at line NNN) is located in the file +simulate.sh and (since the path `simulate.sh` is specified with no +directory components) Swift expects that the `simulate.sh` executable +will be available in your $PATH. + +//// + +Part 2: Running an ensemble of many apps in parallel with a "foreach" loop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `p2.swift` script introduces the `foreach` parallel iteration +construct to run many concurrent simulations. + +image::part02.png[align="center"] + +.p2.swift +----- +sys::[cat ../part02/p2.swift] +----- + +The script also shows an +example of naming the output files of an ensemble run. In this case, the output files will be named +`output/sim_N.out`. + +In part 2, we update the swift.properties file. + +Instead of using shell script (simulate is a link to simulate.sh), we use the equivalent +python version (simulate.py). Swift does not need to know anything about the language an +application is written in. The application can be written in Perl, Python, Java, Fortran, +or any other language. + +The swift.properties file also defines properties that control the parallelism involved +in the swift execution. By setting the tasksPerWorker to the desired number of parallel +application invocations it is possible to take advantage of multiple cores in the +processor. The taskThrottle variable is used to limit number of tasks sent to +workers for execution. Both tasksPerWorker and taskThrottle are set to 2 in the tutorial +and could be set to the number of cores available on your machine. + +----- +sys::[cat ../part02/swift.properties] +----- + + +To run the script and view the output: +----- +$ cd ../part02 +$ swift p2.swift +$ ls output +sim_0.out sim_1.out sim_2.out sim_3.out sim_4.out sim_5.out sim_6.out sim_7.out sim_8.out sim_9.out +$ more output/* +:::::::::::::: +output/sim_0.out +:::::::::::::: + 44 +:::::::::::::: +output/sim_1.out +:::::::::::::: + 55 +... +:::::::::::::: +output/sim_9.out +:::::::::::::: + 82 +----- + +Part 3: Analyzing results of a parallel ensemble +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After all the parallel simulations in an ensemble run have completed, +its typically necessary to gather and analyze their results with some +kind of post-processing analysis program or script. p3.swift +introduces such a postprocessing step. In this case, the files created +by all of the parallel runs of `simulation.sh` will be averaged by by +the trivial "analysis application" `stats.sh`: + +image::part03.png[align="center"] + +.p3.swift +---- +sys::[cat ../part03/p3.swift] +---- + +To run: +---- +$ cd part03 +$ swift p3.swift +---- + +Note that in `p3.swift` we expose more of the capabilities of the +`simulate.sh` application to the `simulation()` app function: + +----- +app (file o) simulation (int sim_steps, int sim_range, int sim_values) +{ + simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o); +} +----- + +`p3.swift` also shows how to fetch application-specific values from +the `swift` command line in a Swift script using `arg()` which +accepts a keyword-style argument and its default value: + +----- +int nsim = toInt(arg("nsim","10")); +int steps = toInt(arg("steps","1")); +int range = toInt(arg("range","100")); +int values = toInt(arg("values","5")); +----- + +Now we can specify that more runs should be performed and that each should run for more timesteps, and produce more that one value each, within a specified range, using command line arguments placed after the Swift script name in the form `-parameterName=value`: + +----- +$ swift p3.swift -nsim=3 -steps=10 -values=4 -range=1000000 +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run001 +Progress: Tue, 03 Jun 2014 17:06:58-0500 +Progress: Tue, 03 Jun 2014 17:06:59-0500 Submitted:2 Active:1 +Progress: Tue, 03 Jun 2014 17:07:00-0500 Submitted:1 Active:2 +Progress: Tue, 03 Jun 2014 17:07:01-0500 Active:3 +Progress: Tue, 03 Jun 2014 17:07:09-0500 Active:2 Finished successfully:1 +Progress: Tue, 03 Jun 2014 17:07:10-0500 Active:1 Finished successfully:2 +Final status:Tue, 03 Jun 2014 17:07:11-0500 Finished successfully:4 + +$ ls output/ +average.out sim_0.out sim_1.out sim_2.out +$ more output/* +:::::::::::::: +output/average.out +:::::::::::::: +651368 +:::::::::::::: +output/sim_0.out +:::::::::::::: + 735700 + 886206 + 997391 + 982970 +:::::::::::::: +output/sim_1.out +:::::::::::::: + 260071 + 264195 + 869198 + 933537 +:::::::::::::: +output/sim_2.out +:::::::::::::: + 201806 + 213540 + 527576 + 944233 +----- + +Now try running (`-nsim=`) 100 simulations of (`-steps=`) 1 second each: + +----- +$ swift p3.swift -nsim=100 -steps=1 +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run002 +Progress: Tue, 03 Jun 2014 17:08:05-0500 +Progress: Tue, 03 Jun 2014 17:08:06-0500 Selecting site:79 Submitted:20 Active:1 +Progress: Tue, 03 Jun 2014 17:08:07-0500 Selecting site:78 Submitted:18 Active:3 Finished successfully:1 +Progress: Tue, 03 Jun 2014 17:08:08-0500 Selecting site:75 Submitted:17 Active:4 Finished successfully:4 +Progress: Tue, 03 Jun 2014 17:08:09-0500 Selecting site:71 Submitted:16 Active:5 Finished successfully:8 +Progress: Tue, 03 Jun 2014 17:08:11-0500 Selecting site:66 Submitted:15 Active:6 Finished successfully:13 +Progress: Tue, 03 Jun 2014 17:08:12-0500 Selecting site:60 Submitted:14 Active:7 Finished successfully:19 +Progress: Tue, 03 Jun 2014 17:08:13-0500 Selecting site:53 Submitted:13 Active:8 Finished successfully:26 +Progress: Tue, 03 Jun 2014 17:08:14-0500 Selecting site:45 Submitted:12 Active:9 Finished successfully:34 +Progress: Tue, 03 Jun 2014 17:08:15-0500 Selecting site:36 Submitted:11 Active:10 Finished successfully:43 +Progress: Tue, 03 Jun 2014 17:08:16-0500 Selecting site:26 Submitted:10 Active:11 Finished successfully:53 +Progress: Tue, 03 Jun 2014 17:08:17-0500 Selecting site:15 Active:21 Finished successfully:64 +Progress: Tue, 03 Jun 2014 17:08:18-0500 Active:16 Finished successfully:84 +Progress: Tue, 03 Jun 2014 17:08:19-0500 Active:11 Finished successfully:89 +Final status:Tue, 03 Jun 2014 17:08:19-0500 Finished successfully:101 +----- + +Part 4: Specifying more complex workflow patterns +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +p4.swift expands the workflow pattern of p3.swift to add additional +stages to the workflow. Here, we generate a dynamic seed value that +will be used by all of the simulations, and for each simulation, we +run an pre-processing application to generate a unique "bias +file". This pattern is shown below, followed by the Swift script. + +image::part06.png[align="center"] + +.p4.swift +---- +sys::[cat ../part04/p4.swift] +---- + +Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement: +----- +seedfile = genseed(1); +----- +and on the bias file, computed and then consumed in these two dependent statements: +----- + biasfile = genbias(1000, 20, simulate_script); + (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile); +----- + +To run: +---- +$ cd ../part04 +$ swift p4.swift +---- + +The default parameters result in the following execution log: + +----- +$ swift p4.swift +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run001 +Progress: Thu, 05 Jun 2014 06:01:36-0500 + +*** Script parameters: nsim=10 range=100 num values=10 + +Progress: Thu, 05 Jun 2014 06:01:37-0500 Submitted:8 Active:2 Finished successfully:11 +Progress: Thu, 05 Jun 2014 06:01:38-0500 Submitted:6 Active:2 Finished successfully:13 +Progress: Thu, 05 Jun 2014 06:01:39-0500 Submitted:4 Active:2 Finished successfully:15 +Progress: Thu, 05 Jun 2014 06:01:40-0500 Submitted:2 Active:2 Finished successfully:17 +Progress: Thu, 05 Jun 2014 06:01:41-0500 Active:2 Finished successfully:19 +Final status:Thu, 05 Jun 2014 06:01:41-0500 Finished successfully:22 +----- + +which produces the following output: +----- +$ ls -lrt output +total 264 +-rw-r--r-- 1 p01532 61532 9 Aug 27 19:17 seed.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_9.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_8.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_7.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_6.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_5.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_4.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_3.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_2.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_1.dat +-rw-r--r-- 1 p01532 61532 180 Aug 27 19:17 bias_0.dat +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_9.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_9.log +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_8.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_7.out +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_6.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_6.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_5.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_5.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:17 sim_4.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_4.log +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_1.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_8.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_7.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_3.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_3.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_2.out +-rw-r--r-- 1 p01532 61532 14898 Aug 27 19:18 sim_2.log +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_1.out +-rw-r--r-- 1 p01532 61532 90 Aug 27 19:18 sim_0.out +-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_0.log +-rw-r--r-- 1 p01532 61532 9 Aug 27 19:18 average.out +-rw-r--r-- 1 p01532 61532 14675 Aug 27 19:18 average.log +----- + +Each sim_N.out file is the sum of its bias file plus newly "simulated" random output scaled by 1,000,000: + +----- +$ cat output/bias_0.dat + 302 + 489 + 81 + 582 + 664 + 290 + 839 + 258 + 506 + 310 + 293 + 508 + 88 + 261 + 453 + 187 + 26 + 198 + 402 + 555 + +$ cat output/sim_0.out +64000302 +38000489 +32000081 +12000582 +46000664 +36000290 +35000839 +22000258 +49000506 +75000310 +----- + +We produce 20 values in each bias file. Simulations of less than that +number of values ignore the unneeded number, while simualtions of more +than 20 will use the last bias number for all remoaining values past +20. As an exercise, adjust the code to produce the same number of +bias values as is needed for each simulation. As a further exercise, +modify the script to generate a unique seed value for each simulation, +which is a common practice in ensemble computations. + +Performing larger Swift runs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To test with larger runs, provide suitable command line arguments. +The example below will run 1000 simulations with each simulation +taking 5 seconds. + +----- +$ swift p4.swift -steps=5 -nsim=1000 +----- Added: SwiftTutorials/swift-localhost-tutorial/doc/TODO =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/TODO (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/TODO 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,137 @@ +PLAN + +basics +cloud +uc3 +multisystems +(modis) + +get hosts.txt for cloud dynamically + +Show variety of UC3 activity and what ran where + +get hostname and app_resources into provider staging (hostname into all _swiftwrap) + +Show gridftp and go connect to process the modis dataset + +Show SwiftR for R users, SwiftPy for Enthought PY? + +fetching apps vs sending apps vs parrot/cvmfs/oasis etc ???? + +Demo script-aps, bin apps, building apps, caching apps. etc. + +Do a Swift BLAST workflow -- with viz !!! (spice it up with help from Dina) + +OSG page from Mats: "Is your job HTC-ready" + +Use new Trunk config in tutorial? + + +Add viz and status display; add node probing to tutorial. + +Debug failure with pinfiles + + +What "exercises to suggest?" + +- run stats app locally +- installed vs transferred apps +- transferred programs vs transferred scripts +- summarize things about the env from the .log files, ala stats. + +Y-style dag: 2 sims and average + +Show how to find temp files at runtime... + +Show how to find hung processes at runtime... (or get other trace data...) + +make your owb versions of the programs; verify that you can substitute them + +Add a layer to isolate shell version v installed version + +Show how to handle a paramfile + +Test if OSGCOnnect works and account string (PorjectName) works OK for both! + +SHow how to handle long lists of files (eg writeData and other swift.properties) + +Expand arith to arb prec using bc + + +ISSUES + +error in pin files? debug and turn back on +PATH issues for various providers + +bin dir? + +setup.csh and .csh testing + +IMPROVEMENTS + +1. Cleanup.sh mentioned in the tutorial +[fixed] + +2. cd ../part02 +[fixed] + +3. support for zsh, csh +[pending] + +4. What results are created and where did the outputs go ? +and pointers to what the log files mean +[pending] + +5. Part4, +- cd ../part04 is missing. Cleanup the README +[fixed] + +6. Part5 +- cd ../part05 is missing. Cleanup +[fixed] + +7. Part7 +-Change from running on Tukey analysis cluster compute nodes +[fixed] + +8. Part6 +command line args for nsim and steps, tell user the different behavior +that can be expected with different arg options +[pending] + +9. Draw workflow for the modis demos +[pending] + +10. Link to more technical swift-lang references +[pending] + +11. Links to detailed language constructs, when explaining +apps, link to the section on apps in the userguide. +[pending] + + + +--- + +From Yadu: + +I've committed all the changes that I made on Friday and you can get them +from -> https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/CIC_2013-08-09 + +I'm trying to make this tutorial easily work on several sites so that, we +do not have to tweak things for every single tutorial. Once this is done, +I can also add it to the test-battery as well. I think this has value +considering that we spend atleast 1-2 days for every demo. + +Now, the major pending item is the Modis demo, specifically the issue of +bringing data to any site and scripts. Slight tweaks to allow for sending +the scripts as args to apps would also be needed. + +So the major changes are: +1. Source the setup.sh script with the target site name +eg. source setups.sh +2. No separate cloud folder +3. [pending] fix modis data and scripts +4. [pending] support for zsh and other shells? + +--- Added: SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/activeplot.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/asciidoc.css 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,26 @@ +a:link { color:navy; } +a:visited { color:navy; } + +.monospaced, code, pre { + font-family: "Courier New", Courier, monospace; + font-size: medium; /* inherit; */ + color: black; + padding: 0; + margin: 0; +} + +/* + background: #f8f8f8; + border: 1px solid #dddddd; + border-left: 5px solid #f0f0f0; + margin-right: 10%; +*/ + +div.listingblock > div.content { + padding: 0.5em; + background: none; + border: none; + border-left: none; + margin-right: none; +} + Added: SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +#!/bin/bash -e + +asciidoc -a icons -a toc -a toplevels=2 -a stylesheet=$PWD/asciidoc.css -a max-width=800px -o tutorial.html README + Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/build_docs.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/cumulativeplot.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.dia ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/figs/modis.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/caution.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/example.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/home.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/important.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/next.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/note.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/prev.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/tip.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/up.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/images/icons/warning.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part01.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part01.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part02.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part02.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part03.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part03.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part04.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part04.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part05.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part05.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/part06.png =================================================================== (Binary files differ) Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/part06.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Added: SwiftTutorials/swift-localhost-tutorial/doc/push.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/push.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/push.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,11 @@ +#! /bin/sh + +# scp -r swift-cray-tutorial.html images *png login.ci.uchicago.edu:/ci/www/projects/swift/tutorials/cray + +pushd . +cd ../../ +tar -cvzf swift-localhost-tutorial.tar.gz swift-localhost-tutorial +mv swift-localhost-tutorial.tar.gz swift-localhost-tutorial/doc/ +popd + +tar zcf - --exclude-vcs *tar.gz *html *png images | ssh login.ci.uchicago.edu "cd /ci/www/projects/swift/tutorials/localhost; tar zxf -" Property changes on: SwiftTutorials/swift-localhost-tutorial/doc/push.sh ___________________________________________________________________ Added: svn:executable + * Added: SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/raven.1000.progress.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,34 @@ +p01532 at raven:~/swift-cray-tutorial/part04> swift p4.swift -nsim=1000 -steps=5 +Swift 0.94.1 RC2 swift-r6895 cog-r3765 + +RunID: 20130824-2131-g43l2g51 +Progress: time: Sat, 24 Aug 2013 21:31:42 -0500 + +*** Script parameters: nsim=1000 steps=5 range=100 + +Progress: time: Sat, 24 Aug 2013 21:31:43 -0500 Initializing:998 Selecting site:2 +Progress: time: Sat, 24 Aug 2013 21:31:45 -0500 Submitting:999 Submitted:1 +Progress: time: Sat, 24 Aug 2013 21:31:46 -0500 Stage in:1 Submitted:999 +Progress: time: Sat, 24 Aug 2013 21:31:52 -0500 Submitted:712 Active:287 Stage out:1 +Progress: time: Sat, 24 Aug 2013 21:31:53 -0500 Submitted:682 Active:278 Stage out:8 Finished successfully:32 +Progress: time: Sat, 24 Aug 2013 21:31:54 -0500 Submitted:605 Active:252 Stage out:32 Finished successfully:111 +Progress: time: Sat, 24 Aug 2013 21:31:55 -0500 Submitted:480 Active:264 Stage out:23 Finished successfully:233 +Progress: time: Sat, 24 Aug 2013 21:31:57 -0500 Submitted:442 Active:287 Stage out:1 Finished successfully:270 +Progress: time: Sat, 24 Aug 2013 21:31:58 -0500 Submitted:432 Active:287 Stage out:1 Finished successfully:280 +Progress: time: Sat, 24 Aug 2013 21:31:59 -0500 Submitted:411 Active:287 Stage out:1 Finished successfully:301 +Progress: time: Sat, 24 Aug 2013 21:32:00 -0500 Submitted:317 Active:287 Finished successfully:396 +Progress: time: Sat, 24 Aug 2013 21:32:01 -0500 Submitted:244 Active:263 Stage out:25 Finished successfully:468 +Progress: time: Sat, 24 Aug 2013 21:32:02 -0500 Submitted:213 Active:233 Stage out:1 Finished successfully:553 +Progress: time: Sat, 24 Aug 2013 21:32:03 -0500 Submitted:210 Active:221 Stage out:1 Finished successfully:568 +Progress: time: Sat, 24 Aug 2013 21:32:04 -0500 Submitted:191 Active:221 Finished successfully:588 +Progress: time: Sat, 24 Aug 2013 21:32:05 -0500 Submitted:122 Active:213 Stage out:7 Finished successfully:658 +Progress: time: Sat, 24 Aug 2013 21:32:06 -0500 Submitted:117 Active:129 Stage out:5 Finished successfully:749 +Progress: time: Sat, 24 Aug 2013 21:32:09 -0500 Submitted:117 Active:98 Stage out:1 Finished successfully:784 +Progress: time: Sat, 24 Aug 2013 21:32:10 -0500 Submitted:96 Active:95 Stage out:1 Finished successfully:808 +Progress: time: Sat, 24 Aug 2013 21:32:11 -0500 Stage in:1 Submitted:33 Active:91 Stage out:4 Finished successfully:871 +Progress: time: Sat, 24 Aug 2013 21:32:12 -0500 Submitted:21 Active:96 Finished successfully:883 +Progress: time: Sat, 24 Aug 2013 21:32:14 -0500 Submitted:21 Active:95 Stage out:1 Finished successfully:883 +Progress: time: Sat, 24 Aug 2013 21:32:15 -0500 Active:93 Stage out:1 Finished successfully:906 +Progress: time: Sat, 24 Aug 2013 21:32:16 -0500 Active:42 Stage out:2 Finished successfully:956 +Progress: time: Sat, 24 Aug 2013 21:32:20 -0500 Active:20 Stage out:1 Finished successfully:979 +Final status: Sat, 24 Aug 2013 21:32:20 -0500 Finished successfully:1000 Added: SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/raven.10000.progress.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,34 @@ +p01532 at raven:~/swift-cray-tutorial/part04> swift p4.swift -nsim=10000 -steps=2 +Swift 0.94.1 RC2 swift-r6895 cog-r3765 + +RunID: 20130824-2132-n4z3dxk1 +Progress: time: Sat, 24 Aug 2013 21:32:43 -0500 + +*** Script parameters: nsim=10000 steps=2 range=100 + +Progress: time: Sat, 24 Aug 2013 21:32:45 -0500 Initializing:3 +Progress: time: Sat, 24 Aug 2013 21:32:46 -0500 Initializing:33 +Progress: time: Sat, 24 Aug 2013 21:32:47 -0500 Initializing:1613 +Progress: time: Sat, 24 Aug 2013 21:32:50 -0500 Selecting site:9999 Submitting:1 +Progress: time: Sat, 24 Aug 2013 21:32:52 -0500 Selecting site:8998 Stage in:1 Submitted:1001 +Progress: time: Sat, 24 Aug 2013 21:32:55 -0500 Selecting site:8998 Submitted:714 Active:287 Stage out:1 +Progress: time: Sat, 24 Aug 2013 21:32:57 -0500 Selecting site:8950 Submitted:714 Active:287 Stage out:1 Finished successfully:48 +Progress: time: Sat, 24 Aug 2013 21:32:58 -0500 Selecting site:8921 Submitting:1 Submitted:713 Active:257 Stage out:31 Finished successfully:77 +Progress: time: Sat, 24 Aug 2013 21:32:59 -0500 Selecting site:8858 Stage in:1 Submitting:5 Submitted:708 Active:178 Stage out:110 Finished successfully:140 +Progress: time: Sat, 24 Aug 2013 21:33:00 -0500 Selecting site:8658 Stage in:1 Submitting:1 Submitted:713 Active:287 Finished successfully:340 +..... +Progress: time: Sat, 24 Aug 2013 21:34:39 -0500 Submitted:555 Active:287 Finished successfully:9158 +Progress: time: Sat, 24 Aug 2013 21:34:40 -0500 Submitted:454 Active:283 Stage out:4 Finished successfully:9259 +Progress: time: Sat, 24 Aug 2013 21:34:41 -0500 Submitted:361 Active:285 Stage out:3 Finished successfully:9351 +Progress: time: Sat, 24 Aug 2013 21:34:42 -0500 Submitted:265 Active:287 Stage out:1 Finished successfully:9447 +Progress: time: Sat, 24 Aug 2013 21:34:43 -0500 Submitted:206 Active:224 Stage out:6 Finished successfully:9564 +Progress: time: Sat, 24 Aug 2013 21:34:44 -0500 Submitted:185 Active:155 Stage out:10 Finished successfully:9650 +Progress: time: Sat, 24 Aug 2013 21:34:45 -0500 Submitted:181 Active:124 Stage out:39 Finished successfully:9656 +Progress: time: Sat, 24 Aug 2013 21:34:46 -0500 Submitted:169 Active:56 Stage out:94 Finished successfully:9681 +Progress: time: Sat, 24 Aug 2013 21:34:47 -0500 Submitted:76 Active:95 Stage out:1 Finished successfully:9828 +Progress: time: Sat, 24 Aug 2013 21:34:48 -0500 Submitted:67 Active:95 Stage out:1 Finished successfully:9837 +Progress: time: Sat, 24 Aug 2013 21:34:49 -0500 Active:81 Stage out:1 Finished successfully:9918 +Progress: time: Sat, 24 Aug 2013 21:34:50 -0500 Active:71 Stage out:1 Finished successfully:9928 +Progress: time: Sat, 24 Aug 2013 21:34:51 -0500 Active:6 Stage out:2 Finished successfully:9992 +Final status: Sat, 24 Aug 2013 21:34:51 -0500 Finished successfully:10000 +p01532 at raven:~/swift-cray-tutorial/part04> Added: SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html =================================================================== --- SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/doc/tutorial.html 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,1518 @@ + + + + + +Swift Tutorial for running on localhost + + + + + +
      +
      +
      +

      This tutorial is viewable at: +http://swift-lang.org/tutorials/localhost/tutorial.html

      +
      +
      +
      +

      Introduction: Why Parallel Scripting?

      +
      +

      Swift is a simple scripting language for executing many instances of +ordinary application programs on distributed parallel resources. +Swift scripts run many copies of ordinary programs concurrently, using +statements like this:

      +
      +
      +
      file out[];
+foreach param,i in paramList {
+  out[i] = simulate(param);
+}
      +
      +

      Swift acts like a structured "shell" language. It runs programs +concurrently as soon as their inputs are available, reducing the need +for complex parallel programming. Swift expresses your workflow in a +portable fashion: The same script runs on multicore computers, +clusters, clouds, grids, and supercomputers.

      +

      In this tutorial, you’ll be able to first try a few Swift examples +on your local machine, to get a sense of the language.

      +
      +
      +
      +

      Swift installation

      +
      +
        +
      1. +

        +Download the file from http://swiftlang.org/packages/swift-0.95-RC6.tar.gz +

        +
        2. +
      2. +

        +Extract by running "tar xfz swift-0.95-RC6.tar.gz" +

        +
        3. +
      3. +

        +Add to PATH by running "export PATH=$PATH:/path/to/swift-0.95-RC6/bin" +

        +
        4. +
      +
      +
      +
      +

      Swift tutorial installation

      +
      +

      Run the following commands to extract these tutorial scripts.

      +
      +
      +
      $ cd $HOME
+$ wget http://swiftlang.org/tutorials/localhost/swift-localhost-tutorial.tar.gz
+$ tar xvfz swift-localhost-tutorial.tar.gz
+$ cd swift-localhost-tutorial
      +
      +

      Run the tutorial setup script

      +
      +
      +
      $ source setup.sh   # You must run this with "source" !
      +
      +
      +

      To check out the tutorial scripts from SVN

      +

      If you later want to get the most recent version of this tutorial from +the Swift Subversion repository, do:

      +
      +
      +
      $ svn co https://svn.ci.uchicago.edu/svn/vdl2/SwiftTutorials/swift-localhost-tutorial  <TODO: Fix link>
      +
      +

      This will create a directory called "swift-localhost-tutorial" which contains all of the +files used in this tutorial.

      +
      +
      +
      +
      +

      Simple "science applications" for the workflow tutorial

      +
      +

      This tutorial is based on two intentionally trivial example programs, +simulation.sh and stats.sh, (implemented as bash shell scripts) +that serve as easy-to-understand proxies for real science +applications. These "programs" behave as follows.

      +
      +

      simulate.sh

      +

      The simulation.sh script serves as a trivial proxy for any more +complex scientific simulation application. It generates and prints a +set of one or more random integers in the range [0-2^62) as controlled +by its command line arguments, which are:

      +
      +
      +
      $ ./app/simulate.sh --help
+./app/simulate.sh: usage:
+    -b|--bias       offset bias: add this integer to all results [0]
+    -B|--biasfile   file of integer biases to add to results [none]
+    -l|--log        generate a log in stderr if not null [y]
+    -n|--nvalues    print this many values per simulation [1]
+    -r|--range      range (limit) of generated results [100]
+    -s|--seed       use this integer [0..32767] as a seed [none]
+    -S|--seedfile   use this file (containing integer seeds [0..32767]) one per line [none]
+    -t|--timesteps  number of simulated "timesteps" in seconds (determines runtime) [1]
+    -x|--scale      scale the results by this integer [1]
+    -h|-?|?|--help  print this help
+$
      +
      +

      All of thess arguments are optional, with default values indicated above as [n].

      +

      With no arguments, simulate.sh prints 1 number in the range of +1-100. Otherwise it generates n numbers of the form (R*scale)+bias +where R is a random integer. By default it logs information about its +execution environment to stderr. Here’s some examples of its usage:

      +
      +
      +
      $ simulate.sh 2>log
+       5
+$ head -4 log
+
+Called as: /home/wilde/swift/tut/CIC_2013-08-09/app/simulate.sh:
+Start time: Thu Aug 22 12:40:24 CDT 2013
+Running on node: login01.osgconnect.net
+
+$ simulate.sh -n 4 -r 1000000 2>log
+  239454
+  386702
+   13849
+  873526
+
+$ simulate.sh -n 3 -r 1000000 -x 100 2>log
+ 6643700
+62182300
+ 5230600
+
+$ simulate.sh -n 2 -r 1000 -x 1000 2>log
+  565000
+  636000
+
+$ time simulate.sh -n 2 -r 1000 -x 1000 -t 3 2>log
+  336000
+  320000
+real    0m3.012s
+user    0m0.005s
+sys     0m0.006s
      +
      +
      +
      +

      stats.sh

      +

      The stats.sh script serves as a trivial model of an "analysis" +program. It reads N files each containing M integers and simply prints +the\ average of all those numbers to stdout. Similarly to simulate.sh +it logs environmental information to the stderr.

      +
      +
      +
      $ ls f*
+f1  f2  f3  f4
+
+$ cat f*
+25
+60
+40
+75
+
+$ stats.sh f* 2>log
+50
      +
      +
      +
      +
      +
      +

      Basics of the Swift language with local execution

      +
      +
      +

      A Summary of Swift in a nutshell

      +
        +
      • +

        +Swift scripts are text files ending in .swift The swift command +runs on any host, and executes these scripts. swift is a Java +application, which you can install almost anywhere. On Linux, just +unpack the distribution tar file and add its bin/ directory to +your PATH. +

        +
        • +
      • +

        +Swift scripts run ordinary applications, just like shell scripts +do. Swift makes it easy to run these applications on parallel and +remote computers (from laptops to supercomputers). If you can ssh to +the system, Swift can likely run applications there. +

        +
        • +
      • +

        +The details of where to run applications and how to get files back +and forth are described in configuration files separate from your +program. Swift speaks ssh, PBS, Condor, SLURM, LSF, SGE, Cobalt, and +Globus to run applications, and scp, http, ftp, and GridFTP to move +data. +

        +
        • +
      • +

        +The Swift language has 5 main data types: boolean, int, +string, float, and file. Collections of these are dynamic, +sparse arrays of arbitrary dimension and structures of scalars and/or +arrays defined by the type declaration. +

        +
        • +
      • +

        +Swift file variables are "mapped" to external files. Swift sends +files to and from remote systems for you automatically. +

        +
        • +
      • +

        +Swift variables are "single assignment": once you set them you can’t +change them (in a given block of code). This makes Swift a natural, +"parallel data flow" language. This programming model keeps your +workflow scripts simple and easy to write and understand. +

        +
        • +
      • +

        +Swift lets you define functions to "wrap" application programs, and +to cleanly structure more complex scripts. Swift app functions take +files and parameters as inputs and return files as outputs. +

        +
        • +
      • +

        +A compact set of built-in functions for string and file +manipulation, type conversions, high level IO, etc. is provided. +Swift’s equivalent of printf() is tracef(), with limited and +slightly different format codes. +

        +
        • +
      • +

        +Swift’s foreach {} statement is the main parallel workhorse of the +language, and executes all iterations of the loop concurrently. The +actual number of parallel tasks executed is based on available +resources and settable "throttles". +

        +
        • +
      • +

        +In fact, Swift conceptually executes all the statements, +expressions and function calls in your program in parallel, based on +data flow. These are similarly throttled based on available resources +and settings. +

        +
        • +
      • +

        +Swift also has if and switch statements for conditional +execution. These are seldom needed in simple workflows but they enable +very dynamic workflow patterns to be specified. +

        +
        • +
      +

      We’ll see many of these points in action in the examples below. Lets +get started!

      +
      +
      +

      Part 1: Run a single application under Swift

      +

      The first swift script, p1.swift, runs simulate.sh to generate a +single random number. It writes the number to a file.

      +
      +
      +p1 workflow +
      +
      +
      +
      p1.swift
      +
      +
      type file;
+
+app (file o) simulation ()
+{
+  simulate stdout=filename(o);
+}
+
+file f <"sim.out">;
+f = simulation();
      +
      +

      To run this script, run the following command:

      +
      +
      +
      $ cd part01
+$ swift p1.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 15:01:28-0500
+Final status:Tue, 03 Jun 2014 15:01:29-0500  Finished successfully:1
+$ cat sim.out
+      84
+$ swift p1.swift
+$ cat sim.out
+      36
      +
      +

      To cleanup the directory and remove all outputs (including the log +files and directories that Swift generates), run the cleanup script +which is located in the tutorial PATH:

      +
      +
      +
      $ cleanup
      +
      +
      + + + +
      +Note +You’ll also find a Swift configuration files in each partNN +directory of this tutorial. These specify the environment-specific +details of where to find application programs and where to run them . +These files will be explained in more detail in parts 4-6, and can +be ignored for now.
      +
      +
      +
      +

      Part 2: Running an ensemble of many apps in parallel with a "foreach" loop

      +

      The p2.swift script introduces the foreach parallel iteration +construct to run many concurrent simulations.

      +
      +
      +part02.png +
      +
      +
      +
      p2.swift
      +
      +
      type file;
+
+app (file o) simulation ()
+{
+  simulate stdout=filename(o);
+}
+
+foreach i in [0:9] {
+  file f <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+  f = simulation();
+}
      +
      +

      The script also shows an +example of naming the output files of an ensemble run. In this case, the output files will be named +output/sim_N.out.

      +

      In part 2, we update the swift.properties file.

      +

      Instead of using shell script (simulate is a link to simulate.sh), we use the equivalent +python version (simulate.py). Swift does not need to know anything about the language an +application is written in. The application can be written in Perl, Python, Java, Fortran, +or any other language.

      +

      The swift.properties file also defines properties that control the parallelism involved +in the swift execution. By setting the tasksPerWorker to the desired number of parallel +application invocations it is possible to take advantage of multiple cores in the +processor. The taskThrottle variable is used to limit number of tasks sent to +workers for execution. Both tasksPerWorker and taskThrottle are set to 2 in the tutorial +and could be set to the number of cores available on your machine.

      +
      +
      +
      site=localhost
+use.provider.staging=true
+execution.retries=2
+
+site.localhost {
+   jobmanager=local
+   filesystem=local
+   initialscore=10000
+   taskWalltime=00:02:00
+   workdir=/tmp/swiftwork
+   tasksPerWorker=2
+   taskThrottle=2
+}
+
+app.localhost.simulate=simulate.py
+app.localhost.stats=stats.py
      +
      +

      To run the script and view the output:

      +
      +
      +
      $ cd ../part02
+$ swift p2.swift
+$ ls output
+sim_0.out  sim_1.out  sim_2.out  sim_3.out  sim_4.out  sim_5.out  sim_6.out  sim_7.out  sim_8.out  sim_9.out
+$ more output/*
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+      44
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+      55
+...
+::::::::::::::
+output/sim_9.out
+::::::::::::::
+      82
      +
      +
      +
      +

      Part 3: Analyzing results of a parallel ensemble

      +

      After all the parallel simulations in an ensemble run have completed, +its typically necessary to gather and analyze their results with some +kind of post-processing analysis program or script. p3.swift +introduces such a postprocessing step. In this case, the files created +by all of the parallel runs of simulation.sh will be averaged by by +the trivial "analysis application" stats.sh:

      +
      +
      +part03.png +
      +
      +
      +
      p3.swift
      +
      +
      type file;
+
+app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+  simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}
+
+app (file o) analyze (file s[])
+{
+  stats filenames(s) stdout=filename(o);
+}
+
+int nsim   = toInt(arg("nsim","10"));
+int steps  = toInt(arg("steps","1"));
+int range  = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));
+
+file sims[];
+
+foreach i in [0:nsim-1] {
+  file simout <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+  simout = simulation(steps,range,values);
+  sims[i] = simout;
+}
+
+file stats<"output/average.out">;
+stats = analyze(sims);
      +
      +

      To run:

      +
      +
      +
      $ cd part03
+$ swift p3.swift
      +
      +

      Note that in p3.swift we expose more of the capabilities of the +simulate.sh application to the simulation() app function:

      +
      +
      +
      app (file o) simulation (int sim_steps, int sim_range, int sim_values)
+{
+  simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o);
+}
      +
      +

      p3.swift also shows how to fetch application-specific values from +the swift command line in a Swift script using arg() which +accepts a keyword-style argument and its default value:

      +
      +
      +
      int nsim   = toInt(arg("nsim","10"));
+int steps  = toInt(arg("steps","1"));
+int range  = toInt(arg("range","100"));
+int values = toInt(arg("values","5"));
      +
      +

      Now we can specify that more runs should be performed and that each should run for more timesteps, and produce more that one value each, within a specified range, using command line arguments placed after the Swift script name in the form -parameterName=value:

      +
      +
      +
      $ swift p3.swift -nsim=3 -steps=10 -values=4 -range=1000000
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Tue, 03 Jun 2014 17:06:58-0500
+Progress: Tue, 03 Jun 2014 17:06:59-0500  Submitted:2  Active:1
+Progress: Tue, 03 Jun 2014 17:07:00-0500  Submitted:1  Active:2
+Progress: Tue, 03 Jun 2014 17:07:01-0500  Active:3
+Progress: Tue, 03 Jun 2014 17:07:09-0500  Active:2  Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:07:10-0500  Active:1  Finished successfully:2
+Final status:Tue, 03 Jun 2014 17:07:11-0500  Finished successfully:4
+
+$ ls output/
+average.out  sim_0.out  sim_1.out  sim_2.out
+$ more output/*
+::::::::::::::
+output/average.out
+::::::::::::::
+651368
+::::::::::::::
+output/sim_0.out
+::::::::::::::
+  735700
+  886206
+  997391
+  982970
+::::::::::::::
+output/sim_1.out
+::::::::::::::
+  260071
+  264195
+  869198
+  933537
+::::::::::::::
+output/sim_2.out
+::::::::::::::
+  201806
+  213540
+  527576
+  944233
      +
      +

      Now try running (-nsim=) 100 simulations of (-steps=) 1 second each:

      +
      +
      +
      $ swift p3.swift -nsim=100 -steps=1
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run002
+Progress: Tue, 03 Jun 2014 17:08:05-0500
+Progress: Tue, 03 Jun 2014 17:08:06-0500  Selecting site:79  Submitted:20  Active:1
+Progress: Tue, 03 Jun 2014 17:08:07-0500  Selecting site:78  Submitted:18  Active:3  Finished successfully:1
+Progress: Tue, 03 Jun 2014 17:08:08-0500  Selecting site:75  Submitted:17  Active:4  Finished successfully:4
+Progress: Tue, 03 Jun 2014 17:08:09-0500  Selecting site:71  Submitted:16  Active:5  Finished successfully:8
+Progress: Tue, 03 Jun 2014 17:08:11-0500  Selecting site:66  Submitted:15  Active:6  Finished successfully:13
+Progress: Tue, 03 Jun 2014 17:08:12-0500  Selecting site:60  Submitted:14  Active:7  Finished successfully:19
+Progress: Tue, 03 Jun 2014 17:08:13-0500  Selecting site:53  Submitted:13  Active:8  Finished successfully:26
+Progress: Tue, 03 Jun 2014 17:08:14-0500  Selecting site:45  Submitted:12  Active:9  Finished successfully:34
+Progress: Tue, 03 Jun 2014 17:08:15-0500  Selecting site:36  Submitted:11  Active:10  Finished successfully:43
+Progress: Tue, 03 Jun 2014 17:08:16-0500  Selecting site:26  Submitted:10  Active:11  Finished successfully:53
+Progress: Tue, 03 Jun 2014 17:08:17-0500  Selecting site:15  Active:21  Finished successfully:64
+Progress: Tue, 03 Jun 2014 17:08:18-0500  Active:16  Finished successfully:84
+Progress: Tue, 03 Jun 2014 17:08:19-0500  Active:11  Finished successfully:89
+Final status:Tue, 03 Jun 2014 17:08:19-0500  Finished successfully:101
      +
      +
      +
      +

      Part 4: Specifying more complex workflow patterns

      +

      p4.swift expands the workflow pattern of p3.swift to add additional +stages to the workflow. Here, we generate a dynamic seed value that +will be used by all of the simulations, and for each simulation, we +run an pre-processing application to generate a unique "bias +file". This pattern is shown below, followed by the Swift script.

      +
      +
      +part06.png +
      +
      +
      +
      p4.swift
      +
      +
      type file;
+
+# app() functions for application programs to be called:
+
+app (file out) genseed (int nseeds, file seed_script)
+{
+  bash "simulate.sh" "-r" 2000000 "-n" nseeds stdout=@out;
+}
+
+app (file out) genbias (int bias_range, int nvalues, file bias_script)
+{
+  bash "simulate.sh" "-r" bias_range "-n" nvalues stdout=@out;
+}
+
+app (file out, file log) simulation (int timesteps, int sim_range,
+                                     file bias_file, int scale, int sim_count,
+                                     file sim_script, file seed_file)
+{
+  bash "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale
+           "-n" sim_count "-S" @seed_file stdout=@out stderr=@log;
+}
+
+app (file out, file log) analyze (file s[], file stat_script)
+{
+  bash "stats.sh" filenames(s) stdout=@out stderr=@log;
+}
+
+# Command line arguments
+
+int  nsim  = toInt(arg("nsim",   "10"));  # number of simulation programs to run
+int  steps = toInt(arg("steps",  "1"));   # number of timesteps (seconds) per simulation
+int  range = toInt(arg("range",  "100")); # range of the generated random numbers
+int  values = toInt(arg("values", "10"));  # number of values generated per simulation
+
+# Main script and data
+
+file simulate_script <"simulate.sh">;
+file stats_script <"stats.sh">;
+file seedfile <"output/seed.dat">;        # Dynamically generated bias for simulation ensemble
+
+tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values);
+seedfile = genseed(1,simulate_script);
+
+file sims[];                      # Array of files to hold each simulation output
+
+foreach i in [0:nsim-1] {
+  file biasfile <single_file_mapper; file=strcat("output/bias_",i,".dat")>;
+  file simout   <single_file_mapper; file=strcat("output/sim_",i,".out")>;
+  file simlog   <single_file_mapper; file=strcat("output/sim_",i,".log")>;
+  biasfile = genbias(1000, 20, simulate_script);
+  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
+  sims[i] = simout;
+}
+
+file stats_out<"output/average.out">;
+file stats_log<"output/average.log">;
+(stats_out,stats_log) = analyze(sims, stats_script);
      +
      +

      Note that the workflow is based on data flow dependencies: each simulation depends on the seed value, calculated in this statement:

      +
      +
      +
      seedfile = genseed(1);
      +
      +

      and on the bias file, computed and then consumed in these two dependent statements:

      +
      +
      +
        biasfile = genbias(1000, 20, simulate_script);
+  (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile);
      +
      +

      To run:

      +
      +
      +
      $ cd ../part04
+$ swift p4.swift
      +
      +

      The default parameters result in the following execution log:

      +
      +
      +
      $ swift p4.swift
+Swift 0.95 Branch SVN swift-r7903 cog-r3908
+RunID: run001
+Progress: Thu, 05 Jun 2014 06:01:36-0500
+
+*** Script parameters: nsim=10 range=100 num values=10
+
+Progress: Thu, 05 Jun 2014 06:01:37-0500  Submitted:8  Active:2  Finished successfully:11
+Progress: Thu, 05 Jun 2014 06:01:38-0500  Submitted:6  Active:2  Finished successfully:13
+Progress: Thu, 05 Jun 2014 06:01:39-0500  Submitted:4  Active:2  Finished successfully:15
+Progress: Thu, 05 Jun 2014 06:01:40-0500  Submitted:2  Active:2  Finished successfully:17
+Progress: Thu, 05 Jun 2014 06:01:41-0500  Active:2  Finished successfully:19
+Final status:Thu, 05 Jun 2014 06:01:41-0500  Finished successfully:22
      +
      +

      which produces the following output:

      +
      +
      +
      $ ls -lrt output
+total 264
+-rw-r--r-- 1 p01532 61532     9 Aug 27 19:17 seed.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_9.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_8.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_7.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_6.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_5.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_4.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_3.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_2.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_1.dat
+-rw-r--r-- 1 p01532 61532   180 Aug 27 19:17 bias_0.dat
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:17 sim_9.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_9.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_8.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:17 sim_7.out
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:17 sim_6.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_6.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:17 sim_5.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_5.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:17 sim_4.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_4.log
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:17 sim_1.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:18 sim_8.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_7.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:18 sim_3.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_3.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:18 sim_2.out
+-rw-r--r-- 1 p01532 61532 14898 Aug 27 19:18 sim_2.log
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:18 sim_1.out
+-rw-r--r-- 1 p01532 61532    90 Aug 27 19:18 sim_0.out
+-rw-r--r-- 1 p01532 61532 14897 Aug 27 19:18 sim_0.log
+-rw-r--r-- 1 p01532 61532     9 Aug 27 19:18 average.out
+-rw-r--r-- 1 p01532 61532 14675 Aug 27 19:18 average.log
      +
      +

      Each sim_N.out file is the sum of its bias file plus newly "simulated" random output scaled by 1,000,000:

      +
      +
      +
      $ cat output/bias_0.dat
+     302
+     489
+      81
+     582
+     664
+     290
+     839
+     258
+     506
+     310
+     293
+     508
+      88
+     261
+     453
+     187
+      26
+     198
+     402
+     555
+
+$ cat output/sim_0.out
+64000302
+38000489
+32000081
+12000582
+46000664
+36000290
+35000839
+22000258
+49000506
+75000310
      +
      +

      We produce 20 values in each bias file. Simulations of less than that +number of values ignore the unneeded number, while simualtions of more +than 20 will use the last bias number for all remoaining values past +20. As an exercise, adjust the code to produce the same number of +bias values as is needed for each simulation. As a further exercise, +modify the script to generate a unique seed value for each simulation, +which is a common practice in ensemble computations.

      +
      +
      +

      Performing larger Swift runs

      +

      To test with larger runs, provide suitable command line arguments. +The example below will run 1000 simulations with each simulation +taking 5 seconds.

      +
      +
      +
      $ swift p4.swift -steps=5 -nsim=1000
      +
      +
      +
      +
      +
      +
      + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/p1.swift =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/p1.swift (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/p1.swift 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,9 @@ +type file; + +app (file o) simulation () +{ + simulate stdout=filename(o); +} + +file f <"sim.out">; +f = simulation(); Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/apps =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run001/apps (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run001/apps 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +* * * INSTALLED INTEL32::LINUX null Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/cf =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run001/cf (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run001/cf 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,29 @@ +kickstart.enabled=maybe +replication.limit=3 +clustering.queue.delay=4 +caching.algorithm=LRU +wrapperlog.always.transfer=false +throttle.transfers=4 +throttle.host.submit=2 +execution.retries=2 +provenance.log=false +use.wrapper.staging=false +pgraph.node.options=color="seagreen", style="filled" +sites.file=${swift.home}/etc/sites.xml +throttle.file.operations=8 +clustering.min.time=60 +foreach.max.threads=16384 +pgraph.graph.options=splines="compound", rankdir="TB" +tracing.enabled=true +pgraph=false +sitedir.keep=false +throttle.score.job.factor=0.2 +provider.staging.pin.swiftfiles=false +throttle.submit=4 +kickstart.always.transfer=false +replication.min.queue.time=60 +replication.enabled=false +tc.file=${swift.home}/etc/tc.data +use.provider.staging=true +clustering.enabled=false +lazy.errors=false Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run001/run001.log 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,219 @@ +2014-06-05 17:35:15,062-0500 DEBUG Loader setting: sites.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml +2014-06-05 17:35:15,063-0500 DEBUG Loader setting: tc.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps +2014-06-05 17:35:15,063-0500 INFO Loader VERSION Swift 0.95 Branch SVN swift-r7903 cog-r3908 +2014-06-05 17:35:15,063-0500 INFO Loader RUN_ID run001 +2014-06-05 17:35:15,064-0500 INFO Loader ARGUMENTS [-runid, run001, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/run001.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml, -tc.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps, -config, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/cf, p1.swift] +2014-06-05 17:35:15,064-0500 INFO Loader MAX_HEAP 954466304 +2014-06-05 17:35:15,064-0500 INFO Loader GLOBUS_HOSTNAME borgmachine +2014-06-05 17:35:15,064-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/. +2014-06-05 17:35:15,064-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", cac hing.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8} +2014-06-05 17:35:15,064-0500 INFO Loader SITES_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml +2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN SITES: + + + + +10000 + +/tmp/swiftwork + +00:02:00 + + + + + +2014-06-05 17:35:15,065-0500 DEBUG textfiles END SITES: +2014-06-05 17:35:15,065-0500 INFO Loader TC_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps +2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN TC: +* * * INSTALLED INTEL32::LINUX null + + +2014-06-05 17:35:15,065-0500 DEBUG textfiles END TC: +2014-06-05 17:35:15,065-0500 DEBUG textfiles BEGIN SWIFTSCRIPT: +type file; + +app (file o) simulation () +{ + simulate stdout=filename(o); +} + +file f <"sim.out">; +f = simulation(); + + +2014-06-05 17:35:15,065-0500 DEBUG textfiles END SWIFTSCRIPT: +2014-06-05 17:35:15,065-0500 INFO Loader p1.swift: source file is new. Recompiling. +2014-06-05 17:35:15,871-0500 INFO Loader RUN_START +2014-06-05 17:35:15,873-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn +2014-06-05 17:35:15,880-0500 INFO swift Using sites file: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/sites.xml +2014-06-05 17:35:15,898-0500 INFO swift Using tc.data: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run001/apps +2014-06-05 17:35:15,975-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost} +2014-06-05 17:35:16,002-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 17:35:16,003-0500 INFO TRACE Variable declaration, line 8, thread R, f +2014-06-05 17:35:16,010-0500 INFO RuntimeStats$ProgressTicker +2014-06-05 17:35:16,010-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 10912672 +2014-06-05 17:35:16,010-0500 INFO TRACE APPCALL, line 3, thread R, simulation() +2014-06-05 17:35:16,010-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation +2014-06-05 17:35:16,015-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-lieirnrl thread=R host=localhost replicationGroup=null +2014-06-05 17:35:16,016-0500 DEBUG swift JOB_START jobid=simulate-lieirnrl tr=simulate arguments=[] tmpdir=p1-run001/jobs/l/simulate-lieirnrl host=localhost +2014-06-05 17:35:16,024-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024) +2014-06-05 17:35:16,049-0500 INFO LocalService Started local service: 127.0.1.1:58430 +2014-06-05 17:35:16,052-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:57712 +2014-06-05 17:35:16,078-0500 INFO Settings Local contacts: [http://128.135.224.51:60251] +2014-06-05 17:35:16,080-0500 INFO CoasterService Started local service: http://127.0.1.1:60251 +2014-06-05 17:35:16,080-0500 INFO CoasterService Reserving channel for registration +2014-06-05 17:35:16,091-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1 +2014-06-05 17:35:16,091-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1 +2014-06-05 17:35:16,091-0500 INFO CoasterService Sending registration +2014-06-05 17:35:16,093-0500 INFO MetaChannel Trying to re-bind current channel +2014-06-05 17:35:16,094-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send) +2014-06-05 17:35:16,094-0500 INFO CoasterService Registration complete +2014-06-05 17:35:16,094-0500 INFO CoasterService Started coaster service: http://127.0.1.1:51304 +2014-06-05 17:35:16,095-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send) +2014-06-05 17:35:16,103-0500 INFO BlockQueueProcessor Starting... id=0605-3505160 +2014-06-05 17:35:16,103-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send) +2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor +Settings { + slots = 20 + jobsPerNode = 1 + workersPerNode = 1 + nodeGranularity = 1 + allocationStepSize = 0.1 + maxNodes = 2097151 + lowOverallocation = 10.0 + highOverallocation = 1.0 + overallocationDecayFactor = 0.001 + spread = 0.9 + reserve = 60.000s + maxtime = 2147483647 + remoteMonitorEnabled = false + internalHostname = 128.135.224.51 + hookClass = null + workerManager = block + workerLoggingLevel = NONE + workerLoggingDirectory = DEFAULT + ldLibraryPath = null + workerCopies = null + directory = null + useHashBang = null + parallelism = 0.01 + coresPerNode = 1 + perfTraceWorker = false + perfTraceInterval = -1 + attributes = {} + callbackURIs = [http://128.135.224.51:60251] +} + +2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor Jobs in holding queue: 1 +2014-06-05 17:35:16,107-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1 +2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s +2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s) +2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120 +2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080 +2014-06-05 17:35:16,108-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0 +2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1 +2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block +2014-06-05 17:35:16,110-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1 +2014-06-05 17:35:16,110-0500 INFO Block Starting block: workers=1, walltime=1080.000s +2014-06-05 17:35:16,111-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-3505160-000000, cores=1, coresPerWorker=1, walltime=1080 +2014-06-05 17:35:16,111-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send) +2014-06-05 17:35:16,112-0500 INFO BlockTaskSubmitter Queuing block Block 0605-3505160-000000 (1x1080.000s) for submission +2014-06-05 17:35:16,112-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks +2014-06-05 17:35:16,112-0500 INFO BlockTaskSubmitter Submitting block Block 0605-3505160-000000 (1x1080.000s) +2014-06-05 17:35:16,112-0500 INFO ExecutionTaskHandler provider=local +2014-06-05 17:35:16,114-0500 INFO Block Block task status changed: Submitting +2014-06-05 17:35:16,114-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript2254547756894002938.pl http://128.135.224.51:60251 0605-3505160-000000 NOLOGGING +2014-06-05 17:35:16,115-0500 INFO Block Block task status changed: Submitted +2014-06-05 17:35:16,116-0500 INFO Block Block task status changed: Active +2014-06-05 17:35:16,116-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-3505160-000000 +2014-06-05 17:35:16,116-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send) +2014-06-05 17:35:16,174-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown] +2014-06-05 17:35:16,177-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:41290] +2014-06-05 17:35:16,177-0500 INFO LocalTCPService Received registration: blockid = 0605-3505160-000000, url = borgmachine +2014-06-05 17:35:16,178-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:41290] +2014-06-05 17:35:16,178-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-3505160-000000 +2014-06-05 17:35:16,179-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) +2014-06-05 17:35:16,180-0500 DEBUG Cpu worker started: block=0605-3505160-000000 host=borgmachine id=0 +2014-06-05 17:35:16,180-0500 DEBUG Cpu ready for work: block=0605-3505160-000000 id=0 +2014-06-05 17:35:16,180-0500 INFO Block Started CPU 0:1402007716s +2014-06-05 17:35:16,181-0500 INFO Block Started worker 0605-3505160-000000:000000 +2014-06-05 17:35:16,181-0500 DEBUG Cpu requesting work: block=0605-3505160-000000 id=0 Cpus sleeping: 1 +2014-06-05 17:35:16,181-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-3505160-000000-000000] +2014-06-05 17:35:16,181-0500 INFO Cpu 0605-3505160-000000:0 (quality: 1.0) submitting urn:1402007715887-1402007715890-1402007715891: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a +2014-06-05 17:35:16,183-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,187-0500 INFO RequestHandler Handler(tag: 7345, REGISTER) unregistering (send) +2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,188-0500 DEBUG Cpu Stagein workerid=000000 +2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,188-0500 INFO RequestHandler Handler(tag: 7346, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,189-0500 INFO GetFileHandler Handler(tag: 7347, GET) request complete; src=proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 17:35:16,197-0500 INFO GetFileHandler Handler(tag: 7347, GET) reader: PR proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 17:35:16,198-0500 INFO GetFileHandler Handler(tag: 7347, GET) -> 8 +2014-06-05 17:35:16,198-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET) +2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 162788727 +2014-06-05 17:35:16,199-0500 INFO GetFileHandler Handler(tag: 7347, GET) sending length: 6078, 95968411 +2014-06-05 17:35:16,199-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078 +2014-06-05 17:35:16,200-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued) +2014-06-05 17:35:16,200-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET) +2014-06-05 17:35:16,200-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last) +2014-06-05 17:35:16,200-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers +2014-06-05 17:35:16,200-0500 INFO GetFileHandler Handler(tag: 7347, GET) unregistering (last) +2014-06-05 17:35:16,202-0500 DEBUG Cpu Active workerid=0605-3505160-000000:000000 +2014-06-05 17:35:16,202-0500 INFO RequestHandler Handler(tag: 7348, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,202-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,240-0500 DEBUG Cpu Stageout workerid=000000 +2014-06-05 17:35:16,240-0500 INFO RequestHandler Handler(tag: 7349, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,241-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,241-0500 INFO PutFileHandler Handler(tag: 7350, PUT) source: /tmp/swiftwork/p1-run001/jobs/l/simulate-lieirnrl/sim.out +2014-06-05 17:35:16,242-0500 INFO PutFileHandler Handler(tag: 7350, PUT) destination: proxy://u-794c6b3b-1466e2c00f8--7fff-u-794c6b3b-1466e2c00f8--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out +2014-06-05 17:35:16,244-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1 +2014-06-05 17:35:16,245-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data +2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 7350, PUT) -> 11 +2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run001/jobs/l/simulate-lieirnrl/sim.out +2014-06-05 17:35:16,245-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out +2014-06-05 17:35:16,246-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done +2014-06-05 17:35:16,246-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send) +2014-06-05 17:35:16,246-0500 INFO PutFileHandler Handler(tag: 7350, PUT) Transfer done +2014-06-05 17:35:16,246-0500 INFO RequestHandler Handler(tag: 7350, PUT) unregistering (send) +2014-06-05 17:35:16,247-0500 DEBUG Cpu Completed +2014-06-05 17:35:16,247-0500 INFO Cpu 0605-3505160-000000:0 jobTerminated +2014-06-05 17:35:16,247-0500 DEBUG Cpu ready for work: block=0605-3505160-000000 id=0 +2014-06-05 17:35:16,247-0500 DEBUG Cpu requesting work: block=0605-3505160-000000 id=0 Cpus sleeping: 1 +2014-06-05 17:35:16,247-0500 INFO RequestHandler Handler(tag: 7351, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,248-0500 DEBUG Cpu 0605-3505160-000000:0 sleeping +2014-06-05 17:35:16,248-0500 INFO LateBindingScheduler jobs queued: 0 +2014-06-05 17:35:16,248-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send) +2014-06-05 17:35:16,248-0500 DEBUG swift JOB_END jobid=simulate-lieirnrl +2014-06-05 17:35:16,248-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 17:35:16,248-0500 INFO swift END_SUCCESS thread=R tr=simulate +2014-06-05 17:35:16,249-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation +2014-06-05 17:35:16,250-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1 +2014-06-05 17:35:16,250-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 29221392 +2014-06-05 17:35:16,259-0500 DEBUG swift Starting cleanups +2014-06-05 17:35:16,259-0500 INFO swift START cleanups=[] +2014-06-05 17:35:16,260-0500 DEBUG swift Ending cleanups +2014-06-05 17:35:16,260-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 67d3caf - Closed +2014-06-05 17:35:16,263-0500 INFO Loader Swift finished with no errors +2014-06-05 17:35:16,263-0500 INFO ServiceManager Cleaning up... +2014-06-05 17:35:16,264-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:51304 +2014-06-05 17:35:16,264-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send) +2014-06-05 17:35:16,366-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send) +2014-06-05 17:35:16,371-0500 INFO BlockQueueProcessor Shutting down blocks +2014-06-05 17:35:16,371-0500 INFO Block Shutting down block Block 0605-3505160-000000 (1x1080.000s) +2014-06-05 17:35:16,372-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-3505160-000000 +2014-06-05 17:35:16,372-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send) +2014-06-05 17:35:16,373-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true +2014-06-05 17:35:16,373-0500 INFO Block Average utilization: 98.5% +2014-06-05 17:35:16,374-0500 INFO CoasterService Shutdown sequence completed +2014-06-05 17:35:16,374-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-3505160-000000, u=98.5 +2014-06-05 17:35:16,374-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send) +2014-06-05 17:35:16,374-0500 INFO Node Node [borgmachine] 0 shut down successfully +2014-06-05 17:35:16,374-0500 INFO AbstractTCPChannel 0605-3505160-000000-000000: Channel shut down +2014-06-05 17:35:16,375-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3505160-000000-000000]] binding to NullChannel +2014-06-05 17:35:16,376-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-3505160-000000-000000] +2014-06-05 17:35:16,376-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3505160-000000-000000]] binding to NullChannel +2014-06-05 17:35:16,377-0500 INFO AbstractTCPChannel 0605-3505160-000000-000000: Channel terminated +2014-06-05 17:35:16,377-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-3505160-000000 +2014-06-05 17:35:16,377-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send) Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run001/sites.xml 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,12 @@ + + + + +10000 + +/tmp/swiftwork + +00:02:00 + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run001/swift.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run001 +Progress: Thu, 05 Jun 2014 17:35:16-0500 +Final status:Thu, 05 Jun 2014 17:35:16-0500 Finished successfully:1 Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/apps =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run002/apps (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run002/apps 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +* * * INSTALLED INTEL32::LINUX null Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/cf =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run002/cf (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run002/cf 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,29 @@ +clustering.enabled=false +replication.min.queue.time=60 +kickstart.enabled=maybe +pgraph.node.options=color="seagreen", style="filled" +clustering.queue.delay=4 +pgraph.graph.options=splines="compound", rankdir="TB" +use.wrapper.staging=false +replication.limit=3 +clustering.min.time=60 +throttle.submit=4 +replication.enabled=false +sites.file=${swift.home}/etc/sites.xml +foreach.max.threads=16384 +tc.file=${swift.home}/etc/tc.data +kickstart.always.transfer=false +throttle.file.operations=8 +lazy.errors=false +caching.algorithm=LRU +throttle.transfers=4 +throttle.host.submit=2 +throttle.score.job.factor=0.2 +wrapperlog.always.transfer=false +tracing.enabled=true +execution.retries=2 +pgraph=false +provenance.log=false +provider.staging.pin.swiftfiles=false +sitedir.keep=false +use.provider.staging=true Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run002/run002.log 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,220 @@ +2014-06-05 17:39:03,925-0500 DEBUG Loader setting: sites.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml +2014-06-05 17:39:03,928-0500 DEBUG Loader setting: tc.file to: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps +2014-06-05 17:39:03,928-0500 INFO Loader VERSION Swift 0.95 RC6 swift-r7900 cog-r3908 +2014-06-05 17:39:03,928-0500 INFO Loader RUN_ID run002 +2014-06-05 17:39:03,928-0500 INFO Loader ARGUMENTS [-runid, run002, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/run002.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml, -tc.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps, -config, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/cf, p1.swift] +2014-06-05 17:39:03,928-0500 INFO Loader MAX_HEAP 954466304 +2014-06-05 17:39:03,928-0500 INFO Loader GLOBUS_HOSTNAME borgmachine +2014-06-05 17:39:03,928-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/. +2014-06-05 17:39:03,928-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=/home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", cac hing.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm.broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8} +2014-06-05 17:39:03,928-0500 INFO Loader SITES_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml +2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN SITES: + + + + +/tmp/swiftwork + + +00:02:00 +10000 + + + + + +2014-06-05 17:39:03,929-0500 DEBUG textfiles END SITES: +2014-06-05 17:39:03,929-0500 INFO Loader TC_FILE /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps +2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN TC: +* * * INSTALLED INTEL32::LINUX null + + +2014-06-05 17:39:03,929-0500 DEBUG textfiles END TC: +2014-06-05 17:39:03,929-0500 DEBUG textfiles BEGIN SWIFTSCRIPT: +type file; + +app (file o) simulation () +{ + simulate stdout=filename(o); +} + +file f <"sim.out">; +f = simulation(); + + +2014-06-05 17:39:03,929-0500 DEBUG textfiles END SWIFTSCRIPT: +2014-06-05 17:39:03,929-0500 INFO Loader p1.swift: source file is new. Recompiling. +2014-06-05 17:39:04,806-0500 INFO Loader RUN_START +2014-06-05 17:39:04,808-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-on-cloud/swift-0.95-RC6 +2014-06-05 17:39:04,814-0500 INFO swift Using sites file: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/sites.xml +2014-06-05 17:39:04,832-0500 INFO swift Using tc.data: /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run002/apps +2014-06-05 17:39:04,906-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost} +2014-06-05 17:39:04,944-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 17:39:04,944-0500 INFO TRACE VARIABLE, line 8, thread R, f MAPPING $, file://localhost/sim.out +2014-06-05 17:39:04,944-0500 INFO TRACE Variable declaration, line 8, thread R, f +2014-06-05 17:39:04,951-0500 INFO RuntimeStats$ProgressTicker +2014-06-05 17:39:04,951-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 9549896 +2014-06-05 17:39:04,951-0500 INFO TRACE APPCALL, line 3, thread R, simulation() +2014-06-05 17:39:04,951-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation +2014-06-05 17:39:04,957-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-2eqnrnrl thread=R host=localhost replicationGroup=null +2014-06-05 17:39:04,957-0500 DEBUG swift JOB_START jobid=simulate-2eqnrnrl tr=simulate arguments=[] tmpdir=p1-run002/jobs/2/simulate-2eqnrnrl host=localhost +2014-06-05 17:39:04,965-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024) +2014-06-05 17:39:04,994-0500 INFO LocalService Started local service: 127.0.1.1:40133 +2014-06-05 17:39:04,998-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:42442 +2014-06-05 17:39:05,022-0500 INFO Settings Local contacts: [http://128.135.224.51:37413] +2014-06-05 17:39:05,024-0500 INFO CoasterService Started local service: http://127.0.1.1:37413 +2014-06-05 17:39:05,024-0500 INFO CoasterService Reserving channel for registration +2014-06-05 17:39:05,035-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1 +2014-06-05 17:39:05,035-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1 +2014-06-05 17:39:05,035-0500 INFO CoasterService Sending registration +2014-06-05 17:39:05,037-0500 INFO MetaChannel Trying to re-bind current channel +2014-06-05 17:39:05,038-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send) +2014-06-05 17:39:05,038-0500 INFO CoasterService Registration complete +2014-06-05 17:39:05,038-0500 INFO CoasterService Started coaster service: http://127.0.1.1:45430 +2014-06-05 17:39:05,038-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send) +2014-06-05 17:39:05,046-0500 INFO BlockQueueProcessor Starting... id=0605-3905050 +2014-06-05 17:39:05,047-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send) +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor +Settings { + slots = 20 + jobsPerNode = 1 + workersPerNode = 1 + nodeGranularity = 1 + allocationStepSize = 0.1 + maxNodes = 2097151 + lowOverallocation = 10.0 + highOverallocation = 1.0 + overallocationDecayFactor = 0.001 + spread = 0.9 + reserve = 60.000s + maxtime = 2147483647 + remoteMonitorEnabled = false + internalHostname = 128.135.224.51 + hookClass = null + workerManager = block + workerLoggingLevel = NONE + workerLoggingDirectory = DEFAULT + ldLibraryPath = null + workerCopies = null + directory = null + useHashBang = null + parallelism = 0.01 + coresPerNode = 1 + perfTraceWorker = false + perfTraceInterval = -1 + attributes = {} + callbackURIs = [http://128.135.224.51:37413] +} + +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Jobs in holding queue: 1 +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1 +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s) +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120 +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080 +2014-06-05 17:39:05,051-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0 +2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1 +2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block +2014-06-05 17:39:05,053-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1 +2014-06-05 17:39:05,053-0500 INFO Block Starting block: workers=1, walltime=1080.000s +2014-06-05 17:39:05,055-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-3905050-000000, cores=1, coresPerWorker=1, walltime=1080 +2014-06-05 17:39:05,055-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send) +2014-06-05 17:39:05,055-0500 INFO BlockTaskSubmitter Queuing block Block 0605-3905050-000000 (1x1080.000s) for submission +2014-06-05 17:39:05,055-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks +2014-06-05 17:39:05,056-0500 INFO BlockTaskSubmitter Submitting block Block 0605-3905050-000000 (1x1080.000s) +2014-06-05 17:39:05,056-0500 INFO ExecutionTaskHandler provider=local +2014-06-05 17:39:05,058-0500 INFO Block Block task status changed: Submitting +2014-06-05 17:39:05,058-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript4102113601101877218.pl http://128.135.224.51:37413 0605-3905050-000000 NOLOGGING +2014-06-05 17:39:05,059-0500 INFO Block Block task status changed: Submitted +2014-06-05 17:39:05,060-0500 INFO Block Block task status changed: Active +2014-06-05 17:39:05,060-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-3905050-000000 +2014-06-05 17:39:05,060-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send) +2014-06-05 17:39:05,102-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown] +2014-06-05 17:39:05,105-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:44286] +2014-06-05 17:39:05,106-0500 INFO LocalTCPService Received registration: blockid = 0605-3905050-000000, url = borgmachine +2014-06-05 17:39:05,106-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:44286] +2014-06-05 17:39:05,107-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-3905050-000000 +2014-06-05 17:39:05,107-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) +2014-06-05 17:39:05,108-0500 DEBUG Cpu worker started: block=0605-3905050-000000 host=borgmachine id=0 +2014-06-05 17:39:05,108-0500 DEBUG Cpu ready for work: block=0605-3905050-000000 id=0 +2014-06-05 17:39:05,109-0500 INFO Block Started CPU 0:1402007945s +2014-06-05 17:39:05,109-0500 INFO Block Started worker 0605-3905050-000000:000000 +2014-06-05 17:39:05,109-0500 DEBUG Cpu requesting work: block=0605-3905050-000000 id=0 Cpus sleeping: 1 +2014-06-05 17:39:05,109-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-3905050-000000-000000] +2014-06-05 17:39:05,110-0500 INFO Cpu 0605-3905050-000000:0 (quality: 1.0) submitting urn:1402007944821-1402007944824-1402007944825: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a +2014-06-05 17:39:05,111-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,112-0500 INFO RequestHandler Handler(tag: 2416, REGISTER) unregistering (send) +2014-06-05 17:39:05,113-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,114-0500 DEBUG Cpu Stagein workerid=000000 +2014-06-05 17:39:05,114-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,114-0500 INFO RequestHandler Handler(tag: 2417, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,115-0500 INFO GetFileHandler Handler(tag: 2418, GET) request complete; src=proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging +2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 2418, GET) reader: PR proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging +2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 2418, GET) -> 8 +2014-06-05 17:39:05,123-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-on-cloud/swift-0.95-RC6/libexec/_swiftwrap.staging +2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET) +2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 1878503554 +2014-06-05 17:39:05,124-0500 INFO GetFileHandler Handler(tag: 2418, GET) sending length: 6078, 1852137531 +2014-06-05 17:39:05,124-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078 +2014-06-05 17:39:05,125-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued) +2014-06-05 17:39:05,125-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET) +2014-06-05 17:39:05,125-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last) +2014-06-05 17:39:05,125-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers +2014-06-05 17:39:05,125-0500 INFO GetFileHandler Handler(tag: 2418, GET) unregistering (last) +2014-06-05 17:39:05,126-0500 DEBUG Cpu Active workerid=0605-3905050-000000:000000 +2014-06-05 17:39:05,126-0500 INFO RequestHandler Handler(tag: 2419, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,126-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,158-0500 DEBUG Cpu Stageout workerid=000000 +2014-06-05 17:39:05,158-0500 INFO RequestHandler Handler(tag: 2420, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,158-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,160-0500 INFO PutFileHandler Handler(tag: 2421, PUT) source: /tmp/swiftwork/p1-run002/jobs/2/simulate-2eqnrnrl/sim.out +2014-06-05 17:39:05,160-0500 INFO PutFileHandler Handler(tag: 2421, PUT) destination: proxy://u-1e4d6cef-1466e2f7f48--7fff-u-1e4d6cef-1466e2f7f48--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out +2014-06-05 17:39:05,164-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1 +2014-06-05 17:39:05,164-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data +2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 2421, PUT) -> 11 +2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run002/jobs/2/simulate-2eqnrnrl/sim.out +2014-06-05 17:39:05,164-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out +2014-06-05 17:39:05,166-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done +2014-06-05 17:39:05,166-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send) +2014-06-05 17:39:05,166-0500 INFO PutFileHandler Handler(tag: 2421, PUT) Transfer done +2014-06-05 17:39:05,166-0500 INFO RequestHandler Handler(tag: 2421, PUT) unregistering (send) +2014-06-05 17:39:05,167-0500 DEBUG Cpu Completed +2014-06-05 17:39:05,167-0500 INFO Cpu 0605-3905050-000000:0 jobTerminated +2014-06-05 17:39:05,167-0500 DEBUG Cpu ready for work: block=0605-3905050-000000 id=0 +2014-06-05 17:39:05,167-0500 DEBUG Cpu requesting work: block=0605-3905050-000000 id=0 Cpus sleeping: 1 +2014-06-05 17:39:05,167-0500 INFO RequestHandler Handler(tag: 2422, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,167-0500 DEBUG Cpu 0605-3905050-000000:0 sleeping +2014-06-05 17:39:05,167-0500 INFO LateBindingScheduler jobs queued: 0 +2014-06-05 17:39:05,167-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send) +2014-06-05 17:39:05,168-0500 DEBUG swift JOB_END jobid=simulate-2eqnrnrl +2014-06-05 17:39:05,168-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 17:39:05,168-0500 INFO swift END_SUCCESS thread=R tr=simulate +2014-06-05 17:39:05,168-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation +2014-06-05 17:39:05,169-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1 +2014-06-05 17:39:05,169-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 27726864 +2014-06-05 17:39:05,179-0500 DEBUG swift Starting cleanups +2014-06-05 17:39:05,180-0500 INFO swift START cleanups=[] +2014-06-05 17:39:05,180-0500 DEBUG swift Ending cleanups +2014-06-05 17:39:05,180-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 61b8ebe3 - Closed +2014-06-05 17:39:05,182-0500 INFO Loader Swift finished with no errors +2014-06-05 17:39:05,183-0500 INFO ServiceManager Cleaning up... +2014-06-05 17:39:05,183-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:45430 +2014-06-05 17:39:05,183-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send) +2014-06-05 17:39:05,286-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send) +2014-06-05 17:39:05,291-0500 INFO BlockQueueProcessor Shutting down blocks +2014-06-05 17:39:05,291-0500 INFO Block Shutting down block Block 0605-3905050-000000 (1x1080.000s) +2014-06-05 17:39:05,292-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-3905050-000000 +2014-06-05 17:39:05,292-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send) +2014-06-05 17:39:05,292-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true +2014-06-05 17:39:05,293-0500 INFO Block Average utilization: 98.3% +2014-06-05 17:39:05,293-0500 INFO CoasterService Shutdown sequence completed +2014-06-05 17:39:05,293-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-3905050-000000, u=98.3 +2014-06-05 17:39:05,293-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send) +2014-06-05 17:39:05,294-0500 INFO Node Node [borgmachine] 0 shut down successfully +2014-06-05 17:39:05,294-0500 INFO AbstractTCPChannel 0605-3905050-000000-000000: Channel shut down +2014-06-05 17:39:05,295-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3905050-000000-000000]] binding to NullChannel +2014-06-05 17:39:05,295-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-3905050-000000-000000] +2014-06-05 17:39:05,296-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-3905050-000000-000000]] binding to NullChannel +2014-06-05 17:39:05,296-0500 INFO AbstractTCPChannel 0605-3905050-000000-000000: Channel terminated +2014-06-05 17:39:05,297-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-3905050-000000 +2014-06-05 17:39:05,297-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send) Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run002/sites.xml 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,12 @@ + + + + +/tmp/swiftwork + + +00:02:00 +10000 + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run002/swift.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +Swift 0.95 RC6 swift-r7900 cog-r3908 +RunID: run002 +Progress: Thu, 05 Jun 2014 17:39:04-0500 +Final status:Thu, 05 Jun 2014 17:39:05-0500 Finished successfully:1 Added: SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run003/sites.xml 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,12 @@ + + + + + +10000 +/tmp/swiftwork +00:02:00 + + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run003/swift.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +SwiftScript program does not exist: option_siteslist= + +For usage information: swift -help + Added: SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run004/sites.xml 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,12 @@ + + + + + + +/tmp/swiftwork +10000 +00:02:00 + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run004/swift.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +SwiftScript program does not exist: in + +For usage information: swift -help + Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run005/run005.log 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,220 @@ +2014-06-05 19:00:03,520-0500 DEBUG Loader setting: sites.file to: run001/sites.xml +2014-06-05 19:00:03,522-0500 DEBUG Loader setting: tc.file to: run001/apps +2014-06-05 19:00:03,522-0500 INFO Loader VERSION Swift 0.95 Branch SVN swift-r7903 cog-r3908 +2014-06-05 19:00:03,522-0500 INFO Loader RUN_ID run005 +2014-06-05 19:00:03,522-0500 INFO Loader ARGUMENTS [-runid, run005, -logfile, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run005/run005.log, -sites.file, /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/run005/sites.xml, -sites.file, run001/sites.xml, -config, run001/cf, -tc.file, run001/apps, p1.swift] +2014-06-05 19:00:03,522-0500 INFO Loader MAX_HEAP 954466304 +2014-06-05 19:00:03,522-0500 INFO Loader GLOBUS_HOSTNAME borgmachine +2014-06-05 19:00:03,522-0500 INFO Loader CWD /home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/. +2014-06-05 19:00:03,523-0500 DEBUG Loader SWIFT_CONFIGURATION Swift configuration ([run001/cf]): {pgraph.node.options=color="seagreen", style="filled", clustering.min.time=60, clustering.queue.delay=4, sites.file=run001/sites.xml, ticker.prefix=Progress: , provenance.log=false, kickstart.enabled=maybe, tc.file=run001/apps, foreach.max.threads=16384, lazy.errors=false, ticker.date.format=, sitedir.keep=false, throttle.score.job.factor=0.2, kickstart.always.transfer=false, throttle.host.submit=2, replication.limit=3, clustering.enabled=false, pgraph=false, file.gc.enabled=true, replication.min.queue.time=60, wrapper.parameter.mode=args, replication.enabled=false, throttle.transfers=4, wrapperlog.always.transfer=false, throttle.submit=4, pgraph.graph.options=splines="compound", rankdir="TB", caching.algorithm=LRU, use.wrapper.staging=false, provider.staging.pin.swiftfiles=false, tracing.enabled=true, wrapper.invocation.mode=absolute, mapping.checker=on, execution.retries=2, cdm .broadcast.mode=file, status.mode=files, use.provider.staging=true, throttle.file.operations=8} +2014-06-05 19:00:03,523-0500 INFO Loader SITES_FILE run001/sites.xml +2014-06-05 19:00:03,523-0500 DEBUG textfiles BEGIN SITES: + + + + +10000 + +/tmp/swiftwork + +00:02:00 + + + + + +2014-06-05 19:00:03,523-0500 DEBUG textfiles END SITES: +2014-06-05 19:00:03,523-0500 INFO Loader TC_FILE run001/apps +2014-06-05 19:00:03,523-0500 DEBUG textfiles BEGIN TC: +* * * INSTALLED INTEL32::LINUX null + + +2014-06-05 19:00:03,523-0500 DEBUG textfiles END TC: +2014-06-05 19:00:03,524-0500 DEBUG textfiles BEGIN SWIFTSCRIPT: +type file; + +app (file o) simulation () +{ + simulate stdout=filename(o); +} + +file f <"sim.out">; +f = simulation(); + + +2014-06-05 19:00:03,524-0500 DEBUG textfiles END SWIFTSCRIPT: +2014-06-05 19:00:03,524-0500 INFO Loader p1.swift: source file is new. Recompiling. +2014-06-05 19:00:04,290-0500 INFO Loader RUN_START +2014-06-05 19:00:04,292-0500 INFO SwiftExecutor swift.home = /home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn +2014-06-05 19:00:04,298-0500 INFO swift Using sites file: run001/sites.xml +2014-06-05 19:00:04,316-0500 INFO swift Using tc.data: run001/apps +2014-06-05 19:00:04,394-0500 INFO AbstractScheduler Setting resources to: {localhost=localhost} +2014-06-05 19:00:04,425-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 19:00:04,425-0500 INFO TRACE VARIABLE, line 8, thread R, f MAPPING $, file://localhost/sim.out +2014-06-05 19:00:04,426-0500 INFO TRACE Variable declaration, line 8, thread R, f +2014-06-05 19:00:04,433-0500 INFO RuntimeStats$ProgressTicker +2014-06-05 19:00:04,433-0500 INFO TRACE APPCALL, line 3, thread R, simulation() +2014-06-05 19:00:04,433-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 11015880 +2014-06-05 19:00:04,433-0500 DEBUG swift PROCEDURE line=3 thread=R name=simulation +2014-06-05 19:00:04,439-0500 DEBUG swift THREAD_ASSOCIATION jobid=simulate-2c3xunrl thread=R host=localhost replicationGroup=null +2014-06-05 19:00:04,440-0500 DEBUG swift JOB_START jobid=simulate-2c3xunrl tr=simulate arguments=[] tmpdir=p1-run005/jobs/2/simulate-2c3xunrl host=localhost +2014-06-05 19:00:04,448-0500 INFO GlobalSubmitQueue No global submit throttle set. Using default (1024) +2014-06-05 19:00:04,472-0500 INFO LocalService Started local service: 127.0.1.1:49694 +2014-06-05 19:00:04,476-0500 INFO BootstrapService Socket bound. URL is http://borgmachine:35322 +2014-06-05 19:00:04,501-0500 INFO Settings Local contacts: [http://128.135.224.51:59460] +2014-06-05 19:00:04,503-0500 INFO CoasterService Started local service: http://127.0.1.1:59460 +2014-06-05 19:00:04,503-0500 INFO CoasterService Reserving channel for registration +2014-06-05 19:00:04,514-0500 INFO MetaChannel MetaChannel [context: cpipe, boundTo: null] binding to cpipe://1 +2014-06-05 19:00:04,514-0500 INFO MetaChannel MetaChannel [context: spipe, boundTo: null] binding to spipe://1 +2014-06-05 19:00:04,514-0500 INFO CoasterService Sending registration +2014-06-05 19:00:04,517-0500 INFO MetaChannel Trying to re-bind current channel +2014-06-05 19:00:04,517-0500 INFO RequestHandler Handler(tag: 1, REGISTER) unregistering (send) +2014-06-05 19:00:04,518-0500 INFO CoasterService Registration complete +2014-06-05 19:00:04,518-0500 INFO CoasterService Started coaster service: http://127.0.1.1:43560 +2014-06-05 19:00:04,518-0500 INFO RequestHandler Handler(tag: 1, CONFIGSERVICE) unregistering (send) +2014-06-05 19:00:04,526-0500 INFO BlockQueueProcessor Starting... id=0605-0007040 +2014-06-05 19:00:04,526-0500 INFO RequestHandler Handler(tag: 2, SUBMITJOB) unregistering (send) +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor +Settings { + slots = 20 + jobsPerNode = 1 + workersPerNode = 1 + nodeGranularity = 1 + allocationStepSize = 0.1 + maxNodes = 2097151 + lowOverallocation = 10.0 + highOverallocation = 1.0 + overallocationDecayFactor = 0.001 + spread = 0.9 + reserve = 60.000s + maxtime = 2147483647 + remoteMonitorEnabled = false + internalHostname = 128.135.224.51 + hookClass = null + workerManager = block + workerLoggingLevel = NONE + workerLoggingDirectory = DEFAULT + ldLibraryPath = null + workerCopies = null + directory = null + useHashBang = null + parallelism = 0.01 + coresPerNode = 1 + perfTraceWorker = false + perfTraceInterval = -1 + attributes = {} + callbackURIs = [http://128.135.224.51:59460] +} + +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Jobs in holding queue: 1 +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Time estimate for holding queue (seconds): 1 +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Considering: Job(id:0 120.000s) +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Max Walltime (seconds): 120 +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Time estimate (seconds): 1080 +2014-06-05 19:00:04,531-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0 +2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1 +2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor Queued: 1 jobs to new Block +2014-06-05 19:00:04,533-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 1, holding.size(): 1 +2014-06-05 19:00:04,533-0500 INFO Block Starting block: workers=1, walltime=1080.000s +2014-06-05 19:00:04,534-0500 INFO RemoteLogHandler BLOCK_REQUESTED id=0605-0007040-000000, cores=1, coresPerWorker=1, walltime=1080 +2014-06-05 19:00:04,534-0500 INFO RequestHandler Handler(tag: 2, RLOG) unregistering (send) +2014-06-05 19:00:04,535-0500 INFO BlockTaskSubmitter Queuing block Block 0605-0007040-000000 (1x1080.000s) for submission +2014-06-05 19:00:04,535-0500 INFO BlockQueueProcessor Added 1 jobs to new blocks +2014-06-05 19:00:04,535-0500 INFO BlockTaskSubmitter Submitting block Block 0605-0007040-000000 (1x1080.000s) +2014-06-05 19:00:04,535-0500 INFO ExecutionTaskHandler provider=local +2014-06-05 19:00:04,538-0500 INFO Block Block task status changed: Submitting +2014-06-05 19:00:04,538-0500 INFO JobSubmissionTaskHandler Submit: in: / command: /usr/bin/perl /home/yadu/.globus/coasters/cscript2332216852508998008.pl http://128.135.224.51:59460 0605-0007040-000000 NOLOGGING +2014-06-05 19:00:04,539-0500 INFO Block Block task status changed: Submitted +2014-06-05 19:00:04,542-0500 INFO Block Block task status changed: Active +2014-06-05 19:00:04,542-0500 INFO RemoteLogHandler BLOCK_ACTIVE id=0605-0007040-000000 +2014-06-05 19:00:04,542-0500 INFO RequestHandler Handler(tag: 3, RLOG) unregistering (send) +2014-06-05 19:00:04,584-0500 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown] +2014-06-05 19:00:04,587-0500 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 128.135.224.51:35162] +2014-06-05 19:00:04,588-0500 INFO LocalTCPService Received registration: blockid = 0605-0007040-000000, url = borgmachine +2014-06-05 19:00:04,588-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 128.135.224.51:35162] +2014-06-05 19:00:04,589-0500 INFO RemoteLogHandler WORKER_ACTIVE blockid=0605-0007040-000000 +2014-06-05 19:00:04,589-0500 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) +2014-06-05 19:00:04,590-0500 DEBUG Cpu worker started: block=0605-0007040-000000 host=borgmachine id=0 +2014-06-05 19:00:04,590-0500 DEBUG Cpu ready for work: block=0605-0007040-000000 id=0 +2014-06-05 19:00:04,591-0500 INFO Block Started CPU 0:1402012804s +2014-06-05 19:00:04,591-0500 INFO Block Started worker 0605-0007040-000000:000000 +2014-06-05 19:00:04,591-0500 DEBUG Cpu requesting work: block=0605-0007040-000000 id=0 Cpus sleeping: 1 +2014-06-05 19:00:04,591-0500 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: 0605-0007040-000000-000000] +2014-06-05 19:00:04,591-0500 INFO Cpu 0605-0007040-000000:0 (quality: 1.0) submitting urn:1402012804305-1402012804308-1402012804309: /bin/bash _swiftwrap.staging -e simulate -out sim.out -err stderr.txt -i -d -if -of sim.out -k -cdmfile -status provider -a +2014-06-05 19:00:04,592-0500 INFO RequestHandler Handler(tag: 5, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,594-0500 INFO RequestHandler Handler(tag: 5897, REGISTER) unregistering (send) +2014-06-05 19:00:04,595-0500 INFO RequestHandler Handler(tag: 6, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,596-0500 DEBUG Cpu Stagein workerid=000000 +2014-06-05 19:00:04,596-0500 INFO RequestHandler Handler(tag: 7, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,596-0500 INFO RequestHandler Handler(tag: 5898, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,596-0500 INFO GetFileHandler Handler(tag: 5899, GET) request complete; src=proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 5899, GET) reader: PR proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 5899, GET) -> 8 +2014-06-05 19:00:04,605-0500 INFO GetFileHandler Handler(tag: 8, GET) request complete; src=file://localhost//home/yadu/src/swift-0.95/cog/modules/swift/dist/swift-svn/libexec/_swiftwrap.staging +2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 8, GET) reader: LR-Handler(tag: 8, GET) +2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 8, GET) sending length: 6078, 1878503554 +2014-06-05 19:00:04,606-0500 INFO GetFileHandler Handler(tag: 5899, GET) sending length: 6078, 1852137531 +2014-06-05 19:00:04,607-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) will ask for 1 buffers for a size of 6078 +2014-06-05 19:00:04,608-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) actual allocated buffers 1(0 queued) +2014-06-05 19:00:04,608-0500 INFO LocalIOProvider LR-Handler(tag: 8, GET) rbuf: RB-LR-Handler(tag: 8, GET) +2014-06-05 19:00:04,609-0500 INFO GetFileHandler Handler(tag: 8, GET) unregistering (last) +2014-06-05 19:00:04,609-0500 INFO ReadBuffer RB-LR-Handler(tag: 8, GET) de-allocating 1 buffers +2014-06-05 19:00:04,609-0500 INFO GetFileHandler Handler(tag: 5899, GET) unregistering (last) +2014-06-05 19:00:04,612-0500 DEBUG Cpu Active workerid=0605-0007040-000000:000000 +2014-06-05 19:00:04,612-0500 INFO RequestHandler Handler(tag: 5900, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,612-0500 INFO RequestHandler Handler(tag: 9, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,645-0500 DEBUG Cpu Stageout workerid=000000 +2014-06-05 19:00:04,645-0500 INFO RequestHandler Handler(tag: 5901, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,645-0500 INFO RequestHandler Handler(tag: 10, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,646-0500 INFO PutFileHandler Handler(tag: 5902, PUT) source: /tmp/swiftwork/p1-run005/jobs/2/simulate-2c3xunrl/sim.out +2014-06-05 19:00:04,646-0500 INFO PutFileHandler Handler(tag: 5902, PUT) destination: proxy://u-5cab3f23-1466e79a59f--7fff-u-5cab3f23-1466e79a59f--8000S//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/./sim.out +2014-06-05 19:00:04,649-0500 INFO PutFileCommand Sending Command(tag: 11, PUT) (t) on cpipe://1 +2014-06-05 19:00:04,649-0500 INFO PutFileCommand Command(tag: 11, PUT) (t) sending data +2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 11, PUT) source: /tmp/swiftwork/p1-run005/jobs/2/simulate-2c3xunrl/sim.out +2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 5902, PUT) -> 11 +2014-06-05 19:00:04,649-0500 INFO PutFileHandler Handler(tag: 11, PUT) destination: file://localhost//home/yadu/src/swift-on-cloud/swift-localhost-tutorial/part01/sim.out +2014-06-05 19:00:04,651-0500 INFO PutFileHandler Handler(tag: 11, PUT) Transfer done +2014-06-05 19:00:04,651-0500 INFO RequestHandler Handler(tag: 11, PUT) unregistering (send) +2014-06-05 19:00:04,651-0500 INFO PutFileHandler Handler(tag: 5902, PUT) Transfer done +2014-06-05 19:00:04,651-0500 INFO RequestHandler Handler(tag: 5902, PUT) unregistering (send) +2014-06-05 19:00:04,652-0500 DEBUG Cpu Completed +2014-06-05 19:00:04,652-0500 INFO Cpu 0605-0007040-000000:0 jobTerminated +2014-06-05 19:00:04,652-0500 DEBUG Cpu ready for work: block=0605-0007040-000000 id=0 +2014-06-05 19:00:04,652-0500 DEBUG Cpu requesting work: block=0605-0007040-000000 id=0 Cpus sleeping: 1 +2014-06-05 19:00:04,652-0500 INFO LateBindingScheduler jobs queued: 0 +2014-06-05 19:00:04,652-0500 INFO RequestHandler Handler(tag: 5903, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,652-0500 DEBUG Cpu 0605-0007040-000000:0 sleeping +2014-06-05 19:00:04,653-0500 INFO RequestHandler Handler(tag: 12, JOBSTATUS) unregistering (send) +2014-06-05 19:00:04,653-0500 DEBUG swift JOB_END jobid=simulate-2c3xunrl +2014-06-05 19:00:04,653-0500 INFO TRACE VARIABLE, line 8, thread R, f INITIALIZED +2014-06-05 19:00:04,653-0500 INFO swift END_SUCCESS thread=R tr=simulate +2014-06-05 19:00:04,654-0500 DEBUG swift PROCEDURE_END line=3 thread=R name=simulation +2014-06-05 19:00:04,655-0500 INFO RuntimeStats$ProgressTicker Finished successfully:1 +2014-06-05 19:00:04,655-0500 INFO RuntimeStats$ProgressTicker HeapMax: 954466304, CrtHeap: 156893184, UsedHeap: 29329432 +2014-06-05 19:00:04,664-0500 DEBUG swift Starting cleanups +2014-06-05 19:00:04,665-0500 INFO swift START cleanups=[] +2014-06-05 19:00:04,665-0500 DEBUG swift Ending cleanups +2014-06-05 19:00:04,665-0500 INFO SwiftFunction Cleaning f:file = java.lang.Object at 61b8ebe3 - Closed +2014-06-05 19:00:04,667-0500 INFO Loader Swift finished with no errors +2014-06-05 19:00:04,667-0500 INFO ServiceManager Cleaning up... +2014-06-05 19:00:04,667-0500 INFO ServiceManager Shutting down service at https://127.0.1.1:43560 +2014-06-05 19:00:04,668-0500 INFO RequestHandler Handler(tag: 3, SHUTDOWNSERVICE) unregistering (send) +2014-06-05 19:00:04,769-0500 INFO RequestHandler Handler(tag: 13, UNREGISTER) unregistering (send) +2014-06-05 19:00:04,770-0500 INFO BlockQueueProcessor Shutting down blocks +2014-06-05 19:00:04,770-0500 INFO Block Shutting down block Block 0605-0007040-000000 (1x1080.000s) +2014-06-05 19:00:04,771-0500 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0605-0007040-000000 +2014-06-05 19:00:04,771-0500 INFO RequestHandler Handler(tag: 14, RLOG) unregistering (send) +2014-06-05 19:00:04,771-0500 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true +2014-06-05 19:00:04,771-0500 INFO Block Average utilization: 98.38% +2014-06-05 19:00:04,771-0500 INFO CoasterService Shutdown sequence completed +2014-06-05 19:00:04,771-0500 INFO RemoteLogHandler BLOCK_UTILIZATION id=0605-0007040-000000, u=98.38 +2014-06-05 19:00:04,771-0500 INFO RequestHandler Handler(tag: 15, RLOG) unregistering (send) +2014-06-05 19:00:04,771-0500 INFO Node Node [borgmachine] 0 shut down successfully +2014-06-05 19:00:04,771-0500 INFO AbstractTCPChannel 0605-0007040-000000-000000: Channel shut down +2014-06-05 19:00:04,772-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-0007040-000000-000000]] binding to NullChannel +2014-06-05 19:00:04,772-0500 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 0605-0007040-000000-000000] +2014-06-05 19:00:04,772-0500 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: 0605-0007040-000000-000000]] binding to NullChannel +2014-06-05 19:00:04,772-0500 INFO AbstractTCPChannel 0605-0007040-000000-000000: Channel terminated +2014-06-05 19:00:04,772-0500 INFO RemoteLogHandler WORKER_SHUTDOWN blockid=0605-0007040-000000 +2014-06-05 19:00:04,772-0500 INFO RequestHandler Handler(tag: 16, RLOG) unregistering (send) Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run005/sites.xml 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,12 @@ + + + + +/tmp/swiftwork +00:02:00 +10000 + + + + + Added: SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/run005/swift.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,4 @@ +Swift 0.95 Branch SVN swift-r7903 cog-r3908 +RunID: run005 +Progress: Thu, 05 Jun 2014 19:00:04-0500 +Final status:Thu, 05 Jun 2014 19:00:04-0500 Finished successfully:1 Added: SwiftTutorials/swift-localhost-tutorial/part01/sim.out =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/sim.out (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/sim.out 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ + 93 Added: SwiftTutorials/swift-localhost-tutorial/part01/swift.properties =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part01/swift.properties (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part01/swift.properties 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,11 @@ +site=localhost +use.provider.staging=true +execution.retries=2 + +site.localhost { + jobmanager=local + filesystem=local + initialscore=10000 + taskWalltime=00:02:00 + workdir=/tmp/swiftwork +} Added: SwiftTutorials/swift-localhost-tutorial/part02/p2.swift =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part02/p2.swift (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part02/p2.swift 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,11 @@ +type file; + +app (file o) simulation () +{ + simulate stdout=filename(o); +} + +foreach i in [0:9] { + file f ; + f = simulation(); +} Added: SwiftTutorials/swift-localhost-tutorial/part02/swift.properties =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part02/swift.properties (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part02/swift.properties 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,16 @@ +site=localhost +use.provider.staging=true +execution.retries=2 + +site.localhost { + jobmanager=local + filesystem=local + initialscore=10000 + taskWalltime=00:02:00 + workdir=/tmp/swiftwork + tasksPerWorker=2 + taskThrottle=2 +} + +app.localhost.simulate=simulate.py +app.localhost.stats=stats.py \ No newline at end of file Added: SwiftTutorials/swift-localhost-tutorial/part03/p3.swift =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part03/p3.swift (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part03/p3.swift 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,27 @@ +type file; + +app (file o) simulation (int sim_steps, int sim_range, int sim_values) +{ + simulate "--timesteps" sim_steps "--range" sim_range "--nvalues" sim_values stdout=filename(o); +} + +app (file o) analyze (file s[]) +{ + stats filenames(s) stdout=filename(o); +} + +int nsim = toInt(arg("nsim","10")); +int steps = toInt(arg("steps","1")); +int range = toInt(arg("range","100")); +int values = toInt(arg("values","5")); + +file sims[]; + +foreach i in [0:nsim-1] { + file simout ; + simout = simulation(steps,range,values); + sims[i] = simout; +} + +file stats<"output/average.out">; +stats = analyze(sims); Added: SwiftTutorials/swift-localhost-tutorial/part03/swift.properties =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part03/swift.properties (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part03/swift.properties 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,13 @@ +site=localhost +use.provider.staging=true +execution.retries=2 + +site.localhost { + jobmanager=local + filesystem=local + initialscore=10000 + taskWalltime=00:02:00 + workdir=/tmp/swiftwork + tasksPerWorker=2 + taskThrottle=2 +} Added: SwiftTutorials/swift-localhost-tutorial/part04/p4.swift =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part04/p4.swift (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part04/p4.swift 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,57 @@ +type file; + +# app() functions for application programs to be called: + +app (file out) genseed (int nseeds, file seed_script) +{ + bash "simulate.sh" "-r" 2000000 "-n" nseeds stdout=@out; +} + +app (file out) genbias (int bias_range, int nvalues, file bias_script) +{ + bash "simulate.sh" "-r" bias_range "-n" nvalues stdout=@out; +} + +app (file out, file log) simulation (int timesteps, int sim_range, + file bias_file, int scale, int sim_count, + file sim_script, file seed_file) +{ + bash "simulate.sh" "-t" timesteps "-r" sim_range "-B" @bias_file "-x" scale + "-n" sim_count "-S" @seed_file stdout=@out stderr=@log; +} + +app (file out, file log) analyze (file s[], file stat_script) +{ + bash "stats.sh" filenames(s) stdout=@out stderr=@log; +} + +# Command line arguments + +int nsim = toInt(arg("nsim", "10")); # number of simulation programs to run +int steps = toInt(arg("steps", "1")); # number of timesteps (seconds) per simulation +int range = toInt(arg("range", "100")); # range of the generated random numbers +int values = toInt(arg("values", "10")); # number of values generated per simulation + +# Main script and data + +file simulate_script <"simulate.sh">; +file stats_script <"stats.sh">; +file seedfile <"output/seed.dat">; # Dynamically generated bias for simulation ensemble + +tracef("\n*** Script parameters: nsim=%i range=%i num values=%i\n\n", nsim, range, values); +seedfile = genseed(1,simulate_script); + +file sims[]; # Array of files to hold each simulation output + +foreach i in [0:nsim-1] { + file biasfile ; + file simout ; + file simlog ; + biasfile = genbias(1000, 20, simulate_script); + (simout,simlog) = simulation(steps, range, biasfile, 1000000, values, simulate_script, seedfile); + sims[i] = simout; +} + +file stats_out<"output/average.out">; +file stats_log<"output/average.log">; +(stats_out,stats_log) = analyze(sims, stats_script); Added: SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +link ../app/simulate \ No newline at end of file Property changes on: SwiftTutorials/swift-localhost-tutorial/part04/simulate.sh ___________________________________________________________________ Added: svn:special + * Added: SwiftTutorials/swift-localhost-tutorial/part04/stats.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part04/stats.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part04/stats.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1 @@ +link ../app/stats \ No newline at end of file Property changes on: SwiftTutorials/swift-localhost-tutorial/part04/stats.sh ___________________________________________________________________ Added: svn:special + * Added: SwiftTutorials/swift-localhost-tutorial/part04/swift.properties =================================================================== --- SwiftTutorials/swift-localhost-tutorial/part04/swift.properties (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/part04/swift.properties 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,13 @@ +site=localhost +use.provider.staging=true +execution.retries=2 + +site.localhost { + jobmanager=local + filesystem=local + initialscore=10000 + taskWalltime=00:02:00 + workdir=/tmp/swiftwork + tasksPerWorker=40 + taskThrottle=40 +} Added: SwiftTutorials/swift-localhost-tutorial/setup.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/setup.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/setup.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,34 @@ +# ensure that this script is being sourced +if [ ${BASH_VERSINFO[0]} -gt 2 -a "${BASH_SOURCE[0]}" = "${0}" ] ; then + echo ERROR: script ${BASH_SOURCE[0]} must be executed as: source ${BASH_SOURCE[0]} + exit 1 +fi + +if [ -f $HOME/.swift/swift.properties ]; then + echo "WARNING: Found swift.properties config file in $HOME/.swift/swift.properties" + echo "Recommend removing the $HOME/.swift/swift.properties as older config file may" + echo "result in unknown behavior" +fi + + +# Setting scripts folder to the PATH env var. +TUTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +if [ _$(which cleanup 2>/dev/null) != _$TUTDIR/bin/cleanup ]; then + echo Adding $TUTDIR/bin:$TUTDIR/app: to front of PATH + PATH=$TUTDIR/bin:$TUTDIR/app:$PATH +else + echo Assuming $TUTDIR/bin:$TUTDIR/app: is already at front of PATH +fi + +if [[ $HOSTNAME == headnode* ]] +then + export JAVA=/usr/local/bin/jdk1.7.0_51/bin + export SWIFT=/usr/local/bin/swift-0.95/bin + export PATH=$JAVA:$SWIFT:$PATH +fi + +echo Swift version is $(swift -version) + +return + Added: SwiftTutorials/swift-localhost-tutorial/test_all.sh =================================================================== --- SwiftTutorials/swift-localhost-tutorial/test_all.sh (rev 0) +++ SwiftTutorials/swift-localhost-tutorial/test_all.sh 2014-06-09 22:37:36 UTC (rev 7910) @@ -0,0 +1,16 @@ +#!/bin/bash + +for i in $(seq 1 1 5) +do + pushd . + echo "============================TESTING part0$i===========================" + cd part0$i + swift p$i.swift + if [[ $? == 0 ]] + then + echo "Cleaning up!" + cleanup + fi + echo -e "\n\n" + popd +done Property changes on: SwiftTutorials/swift-localhost-tutorial/test_all.sh ___________________________________________________________________ Added: svn:executable + *