[Swift-commit] r6612 - in branches/faster: . docs/cookbook docs/siteguide docs/userguide
hategan at ci.uchicago.edu
hategan at ci.uchicago.edu
Mon Jul 8 00:54:14 CDT 2013
Author: hategan
Date: 2013-07-08 00:54:14 -0500 (Mon, 08 Jul 2013)
New Revision: 6612
Removed:
branches/faster/docs/cookbook/cookbook-asciidoc.txt
Modified:
branches/faster/
branches/faster/docs/cookbook/cookbook.txt
branches/faster/docs/cookbook/overview
branches/faster/docs/siteguide/futuregrid
branches/faster/docs/userguide/coasters
branches/faster/docs/userguide/howto_tips
branches/faster/docs/userguide/log-processing
Log:
merged 6215-6223 from trunk
Property changes on: branches/faster
___________________________________________________________________
Modified: svn:mergeinfo
- /branches/release-0.93:4761-5122
/trunk:6172,6177,6182,6189-6190,6202-6203,6206-6208
+ /branches/release-0.93:4761-5122
/trunk:6172,6177,6182,6189-6190,6202-6203,6206-6208,6215-6223
Deleted: branches/faster/docs/cookbook/cookbook-asciidoc.txt
===================================================================
--- branches/faster/docs/cookbook/cookbook-asciidoc.txt 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/cookbook/cookbook-asciidoc.txt 2013-07-08 05:54:14 UTC (rev 6612)
@@ -1,20 +0,0 @@
-// To compile use: asciidoc -a toc -n cookbook-asciidoc.txt
-
-:numbered:
-
-Swift Cookbook
-==============
-Swift team <swift-user at ci.uchicago.edu>
-v0.92, March 2011
-
-include::overview[]
-
-include::swift_basics[]
-
-include::coasters[]
-
-include::diverse_infrastructures[]
-
-include::debugging_swift[]
-
-include::log_processing[]
Modified: branches/faster/docs/cookbook/cookbook.txt
===================================================================
--- branches/faster/docs/cookbook/cookbook.txt 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/cookbook/cookbook.txt 2013-07-08 05:54:14 UTC (rev 6612)
@@ -1,3 +1,7 @@
+// To compile use: asciidoc -a toc -n cookbook-asciidoc.txt
+
+:numbered:
+
Swift Cookbook
==============
Swift team <swift-user at ci.uchicago.edu>
Modified: branches/faster/docs/cookbook/overview
===================================================================
--- branches/faster/docs/cookbook/overview 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/cookbook/overview 2013-07-08 05:54:14 UTC (rev 6612)
@@ -5,7 +5,7 @@
infrastructures. The Swift system comprises of SwiftScript language and the
Swift runtime system. For
introductory material, consult the Swift tutorial found
-link:http://www.ci.uchicago.edu/swift/guides/tutorial.php[here].
+link:http://www.ci.uchicago.edu/swift/guides/trunk/tutorial/tutorial.html[here].
-WARNING: Please be warned that the cookbook is under construction and most recipes are not tested rigorously. You are advised to use them at your own risk.
+WARNING: Please be warned that the cookbook is under construction and the recipes are not tested rigorously.
Modified: branches/faster/docs/siteguide/futuregrid
===================================================================
--- branches/faster/docs/siteguide/futuregrid 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/siteguide/futuregrid 2013-07-08 05:54:14 UTC (rev 6612)
@@ -1,10 +1,14 @@
-Futuregrid (x86 cloud)
-----------------------
+Futuregrid Cloud
+----------------
+The NSF-funded FutureGrid cloud is administered by Indiana University. It
+offers a variety of resources via a multitude of interfaces. Currently, it
+offers cloud resources via three different interfaces: Eucalyptus,
+Nimbus (www.nimbusproject.org), and
+OpenStack (www.openstack.org). The total number of resources at
+FutureGrid is close to 5000 CPU cores and 220~TB of storage from more than six
+physical clusters. We use the resources offered by one such cluster via the
+Nebula middleware.
-FutureGrid is a distributed, high-performance test-bed that allows
-scientists to collaboratively develop and test innovative approaches
-to parallel, grid, and cloud computing.
-
More information on futuregrid can be found at https://portal.futuregrid.org/.
Requesting Futuregrid Access
Modified: branches/faster/docs/userguide/coasters
===================================================================
--- branches/faster/docs/userguide/coasters 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/userguide/coasters 2013-07-08 05:54:14 UTC (rev 6612)
@@ -1,30 +1,42 @@
Coasters
--------
+
+Introduction
+~~~~~~~~~~~~
In many applications, Swift performance can be greatly enhanced by the
-use of CoG coasters. CoG coasters provide a low-overhead job submission
+use of coasters. Coasters provide a low-overhead job submission
and file transfer mechanism suited for the execution of short jobs (on
the order of a few seconds) and the transfer of small files (on the
order of a few kilobytes) for which other grid protocols such as GRAM
and GridFTP are poorly suited.
-The coaster mechanism submits a head job using some other execution
+Benefits
+~~~~~~~~
+
+The coaster mechanism submits a pilot job using some other execution
mechanism such as GRAM, and for each worker node that will be used in a
remote cluster, it submits a worker job, again using some other
execution mechanism such as GRAM. Details on the design of the coaster
mechanism can be found here. <http://wiki.cogkit.org/wiki/Coasters>
-The head job manages file transfers and the dispatch of execution jobs
+Mechanism
+~~~~~~~~~
+
+The pilot job manages file transfers and the dispatch of execution jobs
to workers. Much of the overhead associated with other grid protocols
(such as authentication and authorization, and allocation of worker
nodes by the site's local resource manager) is reduced, because that
-overhead is associated with the allocation of a coaster head or coaster
+overhead is associated with the allocation of a coaster pilot or coaster
worker, rather than with every Swift-level procedure invocation;
potentially hundreds or thousands of Swift-level procedure invocations
can be run through a single worker.
-Coasters can be configured for use in two situations: job execution and
-file transfer.
+Coasters can be configured for two purposes: job execution and
+file staging.
+Coasters How-to
+~~~~~~~~~~~~~~~
+
To use for job execution, specify a sites.xml execution element like this:
----
@@ -46,6 +58,7 @@
----
<filesystem provider="coaster" url="gt2://grid.myhost.org" />
----
+
The url parameter should be a pseudo-URI formed with the URI scheme
being the name of the provider to use to submit the coaster head job,
and the hostname portion being the hostname to be used to execute the
Modified: branches/faster/docs/userguide/howto_tips
===================================================================
--- branches/faster/docs/userguide/howto_tips 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/userguide/howto_tips 2013-07-08 05:54:14 UTC (rev 6612)
@@ -39,8 +39,8 @@
This use case allows the user to submit an MPI job to a remote queue.
-In SwiftScript, we make an invocation that does not look any different
-from any other invocation. In the below code, we do not have any input
+Swift makes an invocation that does not look any different
+from any other invocation. In the code below, we do not have any input
files, and have two output files on stdout and stderr:
----
Modified: branches/faster/docs/userguide/log-processing
===================================================================
--- branches/faster/docs/userguide/log-processing 2013-07-08 05:44:24 UTC (rev 6611)
+++ branches/faster/docs/userguide/log-processing 2013-07-08 05:54:14 UTC (rev 6612)
@@ -5,7 +5,6 @@
To properly generate log plots, you must enable VDL/Karajan logging.
TODO:How?
-
You should check the scripts that you intend to use to determine
what log lines they require and ensure that you are generating
those lines via log4j.properties
@@ -18,8 +17,55 @@
--------------------------------------
TODO: Does it work for coasters-based runs only?
+Meaning and interpretation of Swift log messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A Swift log file is typically a text file with the name of the Swift run and
+its timestamp in the filename and an extension ".log". In addition, a ".rlog"
+file is Swift's resume log which is used by Swift when a run is resumed using
+the "-resume" option. The .rlog file is only for Swift's internal purpose and
+not to be interpreted by the user.
+
+Each line in the log file is typically made up of three parts. The first part
+is the timestamp, the second is the type of log message and the third is the
+message itself. The types of log messages follows the java log4j standard types
+of TRACE, DEBUG, INFO, WARN, ERROR and FATAL. This section lists the various
+Swift log messages and explains the meaning and likely interpretation of those
+messages. Please note that the list is not comprehensive at this time. Also
+note that we will ignore the timestamps here.
+
+. _DEBUG Loader arguments: [-sites.file, sites.xml, -config, cf, -tc.file, tc, postproc-gridftp.swift]_
+. _DEBUG Loader Max heap: 5592449024_
+. _DEBUG textfiles BEGIN_
+. _DEBUG Loader kmlversion is_
+. _DEBUG VDL2ExecutionContext Stack dump_
+. _INFO SetFieldValue Set_
+. _INFO get__site STARTCOMPOUND thread=0-8 name=get__site_
+. _INFO vdl:execute START thread=0-8-0 tr=_
+. _INFO GlobalSubmitQueue No global submit throttle set. Using default (1024)_
+. _DEBUG vdl:execute2 THREAD_ASSOCIATION jobid=getsite-ymj72ook thread=0-8-0-1 host=localhost replicationGroup=xmj72ook_
+. _DEBUG vdl:execute2 JOB_START jobid=getsite-ymj72ook tr=getsite arguments=[644] tmpdir=postproc-gridftp-20120319-0942-adf1o1u2/jobs/y/getsite-ymj72ook host=localhost_
+. _INFO GridExec TASK_DEFINITION_
+. _WARN RemoteConfiguration Find: http://140.221.8.62:38260_
+. _INFO AbstractStreamKarajanChannel$Multiplexer Multiplexer 0 started_
+. _INFO AbstractStreamKarajanChannel$Multiplexer (0) Scheduling SC-null for addition_
+. _INFO AbstractStreamKarajanChannel Channel configured_
+. _INFO MetaChannel MetaChannel: 651528505[1478354072: {}] -> null.bind -> SC-null_
+. _INFO ReadBuffer Will ask for 1 buffers for a size of 6070_
+. _INFO ThrottleManager O maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO ThrottleManager mem=113.54 MB, heap=482.88 MB, maxHeap=5.21 GB_
+. _INFO ThrottleManager I maxBuffers=512, crtBuffers=0, allowedTransfers=256, active=0, suspended=0_
+. _INFO PerformanceDiagnosticInputStream [MEM] Heap total: 482.88 MB, Heap used: 118.58 MB_
+. _INFO vdl:execute END_SUCCESS thread=0-8-0 tr=getsite_
+. _INFO WeightedHostScoreScheduler CONTACT_SELECTED host=localhost, score=99.854_
+. _
+
+
+Log plotting
+~~~~~~~~~~~~
+
Normalize event times in the log to the run start time
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Generate the log, assuming the log is titled +swift-run.log+
@@ -30,7 +76,7 @@
Make a basic load plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
. Normalize the log.
. Build up a load data file:
@@ -47,7 +93,7 @@
Make a basic job completion plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
. Normalize the log.
@@ -64,7 +110,7 @@
------------------------------------------
Make a basic Block allocation plot from Coasters Block log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
. Normalize the log.
@@ -81,7 +127,7 @@
------------------------------------------
Make a job runtime distribution plot from Coasters Cpu log lines
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
. Normalize the log.
More information about the Swift-commit
mailing list