[Swift-commit] r4111 - in SwiftApps/SwiftR: Swift Swift/R Swift/man mxtests/swift
noreply at svn.ci.uchicago.edu
noreply at svn.ci.uchicago.edu
Wed Feb 16 16:38:58 CST 2011
Author: tga
Date: 2011-02-16 16:38:58 -0600 (Wed, 16 Feb 2011)
New Revision: 4111
Added:
SwiftApps/SwiftR/Swift/man/swiftExport.Rd
SwiftApps/SwiftR/Swift/man/swiftLibrary.Rd
Modified:
SwiftApps/SwiftR/Swift/DESCRIPTION
SwiftApps/SwiftR/Swift/R/Tests.R
SwiftApps/SwiftR/Swift/man/Swift-package.Rd
SwiftApps/SwiftR/Swift/man/swiftShutdown.Rd
SwiftApps/SwiftR/mxtests/swift/BootstrapParallelBigger.R
Log:
Working towards new release: added in documentation of new features.
Modified: SwiftApps/SwiftR/Swift/DESCRIPTION
===================================================================
--- SwiftApps/SwiftR/Swift/DESCRIPTION 2011-02-16 20:23:28 UTC (rev 4110)
+++ SwiftApps/SwiftR/Swift/DESCRIPTION 2011-02-16 22:38:58 UTC (rev 4111)
@@ -1,8 +1,8 @@
Package: Swift
Type: Package
Title: R interface to Swift parallel scripting languaage
-Version: 0.1.2
-Date: 2010-02-28
+Version: 0.1.3
+Date: 2010-02-16
Author: Michael Wilde
Maintainer: Michael Wilde <wilde at mcs.anl.gov>
Description: Routines to invoke R functions on remote resources through Swift.
Modified: SwiftApps/SwiftR/Swift/R/Tests.R
===================================================================
--- SwiftApps/SwiftR/Swift/R/Tests.R 2011-02-16 20:23:28 UTC (rev 4110)
+++ SwiftApps/SwiftR/Swift/R/Tests.R 2011-02-16 22:38:58 UTC (rev 4111)
@@ -9,7 +9,11 @@
-basicSwiftTest <- function() { swiftTest_1.1() }
+basicSwiftTest <- function() {
+ swiftInit()
+ swiftTest_1.1()
+ swiftShutdown()
+}
swiftTest_1.1 <- function() {
Modified: SwiftApps/SwiftR/Swift/man/Swift-package.Rd
===================================================================
--- SwiftApps/SwiftR/Swift/man/Swift-package.Rd 2011-02-16 20:23:28 UTC (rev 4110)
+++ SwiftApps/SwiftR/Swift/man/Swift-package.Rd 2011-02-16 22:38:58 UTC (rev 4111)
@@ -222,7 +222,6 @@
\preformatted{
> require(Swift)
> options(swift.server="local")
-> swiftInit(cores=4) # start up Swift on your local machine
> basicSwiftTest() # should take about 1 second
> runAllSwiftTests() # should take < 60 seconds
@@ -258,18 +257,15 @@
To run 4 parallel R servers on the local host:
-\verb{options(swift.server="local")}
-\verb{swiftInit(cores=4)}
+\verb{swiftInit(server="local", cores=4)}
To run 4 R servers on each of two hosts that can be reach by ssh:
-\verb{options(swift.server="ssh")}
-\verb{swiftInit(cores="4", hosts="hostname1 hostname2")}
+\verb{swiftInit(server="ssh", cores="4", hosts="hostname1 hostname2")}
To run 8 R servers for 30 minutes on each of 3 nodes of the Merlot cluster, run this on the login host "merlot", using its "serial" queue:
-\verb{options(swift.server="pbsf")}
-\verb{swiftInit(cores=8, nodes=3, time="00:30:00",
+\verb{swiftInit(server="pbsf", cores=8, nodes=3, time="00:30:00",
queue="serial")}
These Swift servers can be started and left running for multiple
@@ -287,7 +283,6 @@
# Start swift local server as above
require(Swift)
-initSwift()
basicSwiftTest()
}
@@ -295,7 +290,7 @@
\preformatted{
require(Swift)
-initSwift()
+# runAllSwiftTests automatically starts up and shuts down a swift server
runAllSwiftTests()
}
@@ -415,14 +410,19 @@
In addition, the environment variable CLASSPATH should not be set.
-Variables set in the initialze script must typically be set in global
-environment ( var <<- value ); These conventions may need to be
-revisted. The initialexpr script is passed in the same saved/loaded R
+Variables set in the initialize script will appear in the global
+environment of the remote workers.
+The initialexpr script is passed in the same saved/loaded R
object as R functions and arguments. Thus if the expr needs to be run
before these objects can be loaded, then an alternate mathod of
intialization needs to be used. (Eg the original method of passing th
einitvar as a separate file).
+A set of libraries to be loaded remotely can be maintained with the
+\code{swiftLibrary} and \code{swiftDetach} calls. In some cases
+it is desirable to send data back and forth using the \code{swiftExport}
+call.
+
All Swift servers do a \verb{require(methods)} call when they
start. Its not clear if this is unversally desired. It was not done by
default in Rserver but seems to be in interactive R.
@@ -439,10 +439,6 @@
The following caveats are high priority on the FIXME list:
-There is no automatic restart yet if swift dies in its server loop. In
-particular, parsing errors, eg on the Swift initialexpr text, can
-cause the R and hence the Swift server to exit
-
Only lapply is implemented (also SwiftApply) - need to see if we can
cut down arg passing overhead for many of the apply() cases.
@@ -452,9 +448,6 @@
There is no asynchronous swiftapply call yet. Each call must complete
before control is returned to the R command loop.
-Only one instance of each server type (i.e., local, pbs, pbsf, ssh)
-can be started at a time.
-
Lower priority issues are:
There is no easy way yet to alter Swift configuration file variables
@@ -484,6 +477,8 @@
\code{\link{swiftInit}}
\code{\link{swiftShutdown}}
\code{\link{swiftapply}}
+\code{\link{swiftLibrary}}
+\code{\link{swiftExport}}
}
\examples{
library(Swift)
Added: SwiftApps/SwiftR/Swift/man/swiftExport.Rd
===================================================================
--- SwiftApps/SwiftR/Swift/man/swiftExport.Rd (rev 0)
+++ SwiftApps/SwiftR/Swift/man/swiftExport.Rd 2011-02-16 22:38:58 UTC (rev 4111)
@@ -0,0 +1,124 @@
+\name{swiftExport}
+\alias{swiftExport}
+\alias{swiftExportAll}
+\alias{swiftRemoveAll}
+%- Also NEED an '\alias' for EACH other topic documented here.
+\title{
+Exporting global data to remove R workspaces.
+}
+\description{
+ \code{swiftExport} provides a mechanism to export data and functions to remote
+ R workspaces. It is mainly useful in the case where a function being
+ used in a swift*apply call references functions or data in the global
+ workspace. \code{swiftExportAll} exports all non-hidden variables in
+ the global environment of the R session (i.e. those shown by an
+ invocation of \code{ls()} at the interactive prompt).
+ \code{swiftRemoveAll} removes all previously exported data from
+ remote R workspaces.
+
+}
+
+\usage{
+swiftExport(..., list = NULL, file = NULL)
+swiftExportAll()
+swiftRemoveAll()
+}
+%- maybe also 'usage' for other objects documented here.
+\arguments{
+ \item{\dots}{
+ Names of data items to export, either as strings or R identifiers.
+}
+ \item{list}{
+ A list of strings which name data items to export.
+}
+ \item{file}{
+ Optional: a file which is on a file system shared between the local
+ R session and remote R workers in which to store the exported data.
+ If not provided, a file will automatically be chosen.
+ A directory can be specified via the swift.exportdir option.
+}
+}
+\details{
+In many cases if a data or a function is referenced within a function
+given to swift*apply it will automatically be transferred when the
+apply call is carried out. This happens if it is a local
+variable in some function, or if the reference refers to something
+defined in a package that is loaded on the remote machines.
+
+However, functions or data in the global environment (e.g. those defined
+directly in an interactive R session, or assignments using the \code{<<-}
+operator are not automatically transferred to remote R workspaces
+when a swift*apply call is carried out.
+
+If you are working with a large data set, it can be more efficient to store
+the data in the global environment and use swiftExport to transfer the
+data. R workers load exported data on demand,
+so if it is referenced in multiple apply calls, the exported data only needs
+t be transferred once.
+
+swiftExport currently assumes that the swift worker processors share a file
+system with the current machine. The directory through which exported data
+is transferred can be specified with the swift.exportdir option.
+
+The set of exported data persists across multiple swift sessions: if you shut
+down a swift session and start a new one with \code{swiftInit}, the set of
+exported data will be the same until you run \code{swiftRemoveAll}.
+
+}
+\value{
+ No values are returned.
+}
+\references{
+\url{http://www.ci.uchicago.edu/swift}
+}
+\author{
+Swift was developed by: Mihael Hategan, Ben Clifford, Justin Wozniak,
+Yong Zhao, Ian Foster, and Michael Wilde with contributions from Sarah
+Kenny, Ioan Raicu, Luiz Gadelha, Allan Espinosa, Zhao Zhang, David
+Kelly, Jon Monette, Glen Hocky, Tom Uram, Wenjun Wu, and other users.
+
+Swift R package developed by Michael Wilde, Tim Armstrong and the OpenMx project
+
+Maintainer: Michael Wilde <wilde at mcs.anl.gov>
+}
+\note{
+%% ~~further notes~~
+}
+
+%% ~Make other sections like Warning with \section{Warning }{....} ~
+
+\seealso{
+\code{\link{Swift-package}}
+}
+\examples{
+
+swiftInit(server="local", cores=2)
+
+# global data
+mydata <- c(1,2,3,4)
+
+getter <- function (i) { mydata[[i]] }
+
+# Won't be able to find mydata
+swiftLapply(1:4, getter)
+
+swiftExport(mydata)
+# or: swiftExport(list=c("mydata"))
+# This should work better!
+swiftLapply(1:4, getter)
+
+swiftRemoveAll()
+# won't work again
+swiftLapply(1:4, getter)
+
+swiftExportAll()
+# works again
+swiftLapply(1:4, getter)
+
+swiftShutdown()
+
+}
+% Add one or more standard keywords, see file 'KEYWORDS' in the
+% R documentation directory.
+\keyword{ ~kwd1 }
+\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
Added: SwiftApps/SwiftR/Swift/man/swiftLibrary.Rd
===================================================================
--- SwiftApps/SwiftR/Swift/man/swiftLibrary.Rd (rev 0)
+++ SwiftApps/SwiftR/Swift/man/swiftLibrary.Rd 2011-02-16 22:38:58 UTC (rev 4111)
@@ -0,0 +1,76 @@
+\name{swiftLibrary}
+\alias{swiftLibrary}
+\alias{swiftDetach}
+\alias{swiftDetachAll}
+\title{
+ Loading and unloading remote Swift libraries.
+}
+\description{
+These functions allow packages to be loaded and unloaded on remote R instances
+managed by SwiftR.
+}
+\usage{
+swiftLibrary(packname)
+swiftDetach(packname)
+swiftDetachAll()
+}
+\arguments{
+ \item{packname}{
+ The name of the package to attach or detach.
+ The arguments accepted are the same as the standard R \code{library}
+ and \code{detach} functions. They can either be provided as R identifiers
+ (e.g. \code{library(Swift)}) or as strings (e.g. \code{library("Swift")}).
+ swiftDetach accepts package names with a "package:" prefix in the same
+ was as \code{detach} (e.g. \code{detach(package:Swift)}).
+}
+}
+\details{
+If a library is added through the \code{swiftLibrary} call,
+it will be loaded for any future swift*apply calls. The library can be unloaded using \code{swiftDetach} or \code{swiftDetachAll}.
+The implementation guarantees that any subsequent swift*apply calls are run in
+R workspaces with the appropriate set of packages loaded.
+
+Note that the package will not be loaded into the current R workspace.
+
+If you pair up \code{swiftLibrary} and
+\code{swiftDetach} calls with the local versions,
+you will have the same libraries loaded on remote R workspaces and in the local
+workspace.
+
+\code{swiftDetachAll} unloads all libraries previously loaded using this
+mechanism.
+}
+\value{
+ No value is returned.
+}
+\references{
+\url{http://www.ci.uchicago.edu/swift}
+}
+\author{
+Swift was developed by: Mihael Hategan, Ben Clifford, Justin Wozniak,
+Yong Zhao, Ian Foster, and Michael Wilde with contributions from Sarah
+Kenny, Ioan Raicu, Luiz Gadelha, Allan Espinosa, Zhao Zhang, David
+Kelly, Jon Monette, Glen Hocky, Tom Uram, Wenjun Wu, and other users.
+
+Swift R package developed by Michael Wilde, Tim Armstrong and the OpenMx project
+
+Maintainer: Michael Wilde <wilde at mcs.anl.gov>
+}
+\note{
+%% ~~further notes~~
+}
+
+%% ~Make other sections like Warning with \section{Warning }{....} ~
+
+\seealso{
+%% ~~objects to See Also as \code{\link{help}}, ~~~
+\code{\link{Swift-package}}
+}
+\examples{
+
+
+}
+% Add one or more standard keywords, see file 'KEYWORDS' in the
+% R documentation directory.
+\keyword{ ~kwd1 }
+\keyword{ ~kwd2 }% __ONLY ONE__ keyword per line
Modified: SwiftApps/SwiftR/Swift/man/swiftShutdown.Rd
===================================================================
--- SwiftApps/SwiftR/Swift/man/swiftShutdown.Rd 2011-02-16 20:23:28 UTC (rev 4110)
+++ SwiftApps/SwiftR/Swift/man/swiftShutdown.Rd 2011-02-16 22:38:58 UTC (rev 4111)
@@ -7,17 +7,20 @@
%% ~~ A concise (1-5 lines) description of what the function does. ~~
}
\usage{
-swiftShutdown(handle = NULL)
+swiftShutdown(handle = NULL, all=FALSE)
}
\arguments{
\item{handle}{
The handle value returned by "\code{swiftInit}" when a worker was
started up. If provided, only that worker will be shut down.
- If handle is not provided, all workers started during this R
- session will be shut down.
+ If handle is not provided and \code{all} is FALSE, then the most
+ recently started worker will be shut down.
}
+ \item{all}{
+ Whether to terminate all workers started within this R session.
}
+}
\details{
%% ~~ If necessary, more details than the description above ~~
}
Modified: SwiftApps/SwiftR/mxtests/swift/BootstrapParallelBigger.R
===================================================================
--- SwiftApps/SwiftR/mxtests/swift/BootstrapParallelBigger.R 2011-02-16 20:23:28 UTC (rev 4110)
+++ SwiftApps/SwiftR/mxtests/swift/BootstrapParallelBigger.R 2011-02-16 22:38:58 UTC (rev 4111)
@@ -53,7 +53,6 @@
list(75, 100, 64),
list(75, 100, 128),
list(75, 100, 256),
- list(75, 100, 256 + 128),
list(75, 100, 512),
list(75, 100, 786),
list(75, 100, 1024),
More information about the Swift-commit
mailing list