[Swift-commit] r3606 - SwiftApps/SwiftR
noreply at svn.ci.uchicago.edu
noreply at svn.ci.uchicago.edu
Tue Sep 7 11:59:21 CDT 2010
Author: wilde
Date: 2010-09-07 11:59:21 -0500 (Tue, 07 Sep 2010)
New Revision: 3606
Added:
SwiftApps/SwiftR/README
Modified:
SwiftApps/SwiftR/TODO
Log:
Update notes.
Added: SwiftApps/SwiftR/README
===================================================================
--- SwiftApps/SwiftR/README (rev 0)
+++ SwiftApps/SwiftR/README 2010-09-07 16:59:21 UTC (rev 3606)
@@ -0,0 +1,132 @@
+TIPS AND REMINDERS FOR THE FORGETFUL DEVELOPER :)
+
+
+* NOTES on where everything lives:
+
+Am testing on PADS
+
+ ~/SwiftR is my "project" main working dir
+
+R is under ~/R and ~/R/pads (compiled for PADS; the ~/R/bin/R
+executable gets a library error on pads)
+
+To build R:
+ http://cran.r-project.org/src/base/R-2/R-2.11.1.tar.gz (or desired release)
+ tar zxf *gz
+ cd ddd
+ ./configure --prefix=/home/wilde/R/R-2.11.0
+ make
+ make install
+
+Need to put the right R in the PATH for building and testing.
+Using R 2.11 as of 2010.0903
+
+OpenMx source tree checked out under: ~/SwiftR/OpenMx
+
+Repo is: https://svn.ci.uchicago.edu/svn/vdl2/SwiftApps/SwiftR
+
+R packages are installed under: ~/RPackages
+(These are the release-independent packages?
+FIXME: how does this relate to packages under the current R relesae?
+
+#-----
+bri$ pwd
+/home/wilde/R/R-2.11.0
+bri$ cd library/
+bri$ pwd
+/home/wilde/R/R-2.11.0/library
+bri$ ls
+KernSmooth/ OpenMx/ base/ cluster/ foreign/ grid/ mgcv/ rpart/ stats/ tcltk/
+MASS/ R.css boot/ codetools/ grDevices/ lattice/ nlme/ spatial/ stats4/ tools/
+Matrix/ Swift/ class/ datasets/ graphics/ methods/ nnet/ splines/ survival/ utils/
+bri$
+#-----
+
+* Note: dont yet know if we do or do not need separate compiles between
+communicado, bridled, pads. teraport, and other systems on the CI
+net. Hopefully not; if we do, weill need to create a tree to R
+releases each with a separate subtree for user-installed packages)
+Seems that we do, at least for PADS, we get this error:
+
+login1$ ~/R/bin/R
+/home/wilde/R/lib64/R/bin/exec/R:
+ error while loading shared libraries: libreadline.so.4:
+ cannot open shared object file: No such file or directory
+login1$
+
+*--
+
+-- Swift package *source* (tbd) is under:
+
+ ~/SwiftR/Swift (the "Swift" package)
+
+-- Swift package is installed under: ????????
+
+* Setup
+
+Get latest R in path for installs etc.
+
+ PATH=/home/wilde/R/R-2.11.0/bin:$PATH
+
+* Fetch, Build, Install OpenMx
+
+* Make Swift Package
+
+
+FIXME: document R CMD BUILD and R CMD INSTALL here
+
+
+ cd ~/SwiftR/Swift?
+
+* How to build swift package into a source archive:
+
+ R CMD build Swift
+
+* package can also be fetched from svn abd built/installed from source I think? Isnt this how OpenMx itself works?
+
+* How to build swift package right into source code
+
+* How to test the package
+
+ R CMD check Swift
+
+* Install Swift Package
+
+ R CMD INSTALL Swift # Puts package under current R release:
+
+ $ R CMD INSTALL Swift | tee INSTALL.out
+
+#--- Example:
+
+login1$ R CMD INSTALL Swift
+* installing to library '/home/wilde/R/R-2.11.0/library'
+* installing *source* package 'Swift' ...
+...etc
+
+#---
+
+ R CMD INSTALL -l ~/RLibrary Swift # Puts package under user lib; then need to add this to R search path?
+ R CMD INSTALL Swift_0.1.tar.gz # Installs from a pre-built source archive instead of from Swift/ source dir
+
+* Run tests Swift package tests manually
+
+$ cd ~/SwiftR/Swift
+$ R
+> require(Swift)
+> source("tests/TestSwift.R")
+
+################## INTERNAL DESIGN LOGIC
+
+the core R function is switapply()
+
+Its a bit more general then the lApply/sApply/etc functions:
+
+For each call it specfies a full set of arguments
+
+lApply calls swiftapply
+
+in swiftapply the remote calls are batched
+
+option processing:
+
+
Modified: SwiftApps/SwiftR/TODO
===================================================================
--- SwiftApps/SwiftR/TODO 2010-09-07 16:54:44 UTC (rev 3605)
+++ SwiftApps/SwiftR/TODO 2010-09-07 16:59:21 UTC (rev 3606)
@@ -1,73 +1,138 @@
+*** TO DO LIST:
-*** NOTES on where everything lives:
-Am testing on PADS
-~/SwiftR is my "project" main working dir
+ need persistent coaster service
-R is under ~/R and ~/R/pads (compiled for PADS; the ~/R/bin/R executable gets a library error on pads)
-OpenMx source tree checked out under: ~/SwiftR/OpenMx
+ output issues from swift worker to clean up (see bottom of this file):
+ "Directory no empty" worker.pl line 1001
+ unlink messages
-R packages are installed under: ~/RPackages
+ clean up of R procs on remote side
---
-(Note: dont yet know if we do or do not need separate compiles between
-communicado, bridles, pads. teraport, and other systems on the CI
-net. Hopefully nit; if we do, weill need to create a tree to R
-releases each with a separate subtree for user-installed packages)
-Seems that we do, at least for PADS, we get this error:
+[] debug version
+ create version of swiftapply etc that runs by saving file, printing
+ message, and waiting for user to do "swiftrun" in a separate
+ window. This will mimic the exec path and enable a developer to debug
+ serially, and under a debugger, on both the client and server side.
+ --> did i do this already?
-login1$ ~/R/bin/R
-/home/wilde/R/lib64/R/bin/exec/R:
- error while loading shared libraries: libreadline.so.4:
- cannot open shared object file: No such file or directory
-login1$
+x check snow/fall code to see if we are missing anything for
+ transparent remote exec
+ (This was list propoerties; fixed now; anything else?)
---
+x n args
+x batch
+x into svn
+x unique dirs
+ clean up source code formatting: Use OpenMx conventions?
+ (spaces, tabbing?)
-Swift package *source* (tbd) is under:
+For OpenMx 1.0:
-~/SwiftR/Swift (the "Swift" package)
+ coasters for persistent R Servers
+ handle entire apply family
+ determine if we should improve the way we handle options (ask OpenMx team)
-Swift package is installed under:
+ use match.fun ( ??? )
+ ensure (and test) that all expected object attributes are passed and preserved on inputs and outputs (names, dims, etc)
+ capture stdout/err, perhaps with length limits, into R return vars
+ control over Swift properties; set defaults to "no retries":
+ Progress: Active:7 Failed but can retry:2 ==> should not happen
+ R docs
+x R package (SwiftR)
+ Swift docs
+ pass the func as val ( ??? )
+ pass extra funcs and packages required
+ pass extra vals
+ pass extra files
+ better download and install of Swift (compliant with CRAN regs)
+ args as alists vs args as list( ??? )
+ return good error messages including messages from R eval and from Swift
+ Name pbs jobs mnemonically
+ options for cleaning up and/or retaining intermidiate save/load and stdout/err files
+ tools for displaying what was sent / received in calls (easy to pick apart from retained Rcall and Rret files)
+_ select sites and swift args (throttles etc)
+ Improve tests - see list below
+
+ runids, output logging
+ select exec sites and swift params etc
+ make polymorphic to *apply and snow
-*** TO DO LIST:
+For General R usage:
-x n args
-x batch
-x into svn
-x unique dirs
-_ select sites and swift args (throttles etc)
- R docs
- R package (SwiftR)
- Swift docs
+ integration with snowfall/snow?
+
async exec
clean up boot: fix all calls to statistics; update boot package w/ pboot() or swiftboot()
error handling and null and missing values: ensure res#s correspond to arg#s
status
specify swift scripts
+ specify data files to be passed to remote side
+ setup the R envs (???)
+ specify initial R code to run on remote side: (if !initialized_swift) { initialRCodeHere }
run async and grab status (track 'runs' in R)
increm result collect
- pass the func as val
- pass extra funcs and packages required
- pass extra vals
- pass extra files
specifiy unique swift scritps ala Dirk's tools
- setup the R envs (???)
- coasters for persistent R Servers
- test suites
use littleR
- args as alists vs args as list
- runids, output logging
- select exec sites and swift params etc
- make polymorphic to *apply and snow
stream results back to R (so use can inspect as they arrive)
- (pull them in with a Swift.poll() func)
- handle discontiguous results
- return good error messages including messages from R eval and from Swift
- TUI
- Name pbs jobs mnemonically
+ (pull them in with a Swift.poll() func)
+ handle discontiguous results (??) and incomplete results (not every element of apply() completes)
+ TUI (in various forms)
+
+*** TESTS NEEDED
+
+more tests
+test with better data that makes it easier to cross check the results
+dont require boot() for the test datasets
+but test (p)boot() if available
+add output data comparisons (or other output checks) to all tests
+test across a bigger range of types and R weirnesses like local funcs, namespaces, non local assigns, closures, data types etc.
+test on matrices
+test on big matrixes etc to assure that we have no length issues
+tests to ensure all data object attrs passed in and out OK
+
+test strategy:
+- can we part data of arbitrary structure back and forth?
+- can we pass functions back and forth?
+- do the remote calls work the same regardless of whether they go to an R server or a pre-call R invocation
+
+flag to control # of tests and degree
+
+----
+
+OUTPUT and SWIFT EXEC ISSUES TO CLEAN UP:
+
+Issues below was that we created a long-lived process that hung on to open files, preventing jobdir from being removed.
+
+bri$ ./TestSwiftScript.sh run03
+swift output is in: swift.stdouterr.7124, pids in swift.workerpids.7124
+Swift svn swift-r3591 cog-r2868 (cog modified locally)
+
+RunID: 20100902-1723-etxm22pg
+Progress:
+Passive queue processor initialized. Callback URI is http://128.135.125.18:50003
+Coaster contact: http://128.135.125.18:50003
+Started workers from these ssh processes: 27297
+logilename: /home/wilde/SwiftR/run03/swiftworkerlogs/worker-2010.0902.172334.09027.log
+Progress: Active:1
+Can't remove directory /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj (Directory not empty) at /home/wilde/swift/lab/worker.pl line 1001
+Final status: Finished successfully:1
+Cleaning up...
+Shutting down service at https://128.135.125.18:50002
+Got channel MetaChannel: 570110481[1922304900: {}] -> null[1922304900: {}]
++ Done
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/_swiftwrap.staging
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/outdir/r.0001.out
+rmdir /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/outdir
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/3
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/stdout.txt
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/wrapper.log
+unlink /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj/cb1
+rmdir /home/wilde/SwiftR/run03/swiftwork/rtest-20100902-1723-etxm22pg-f-runr-f90rr6yj
+Terminating worker processes 27297 and starter 27126
+bri$
More information about the Swift-commit
mailing list