[Swift-commit] r4434 - SwiftApps/SwiftR

[tga at ci.uchicago.edu]

Author: tga
Date: 2011-04-28 11:14:21 -0500 (Thu, 28 Apr 2011)
New Revision: 4434

Modified:
   SwiftApps/SwiftR/IMMEDIATE-TODO
Log:
updating task list.


Modified: SwiftApps/SwiftR/IMMEDIATE-TODO
===================================================================
--- SwiftApps/SwiftR/IMMEDIATE-TODO	2011-04-28 04:34:03 UTC (rev 4433)
+++ SwiftApps/SwiftR/IMMEDIATE-TODO	2011-04-28 16:14:21 UTC (rev 4434)
@@ -6,15 +6,6 @@
 
 
 VERY HIGH:
--- Streamlined tutorial for Multicore:
-    * Simple page with nothing aside from R commands for simplest case
-    * link to installation instructions
-    * link to advanced info
-
-VERY HIGH:
--- OpenMx doco on wiki
-
-VERY HIGH:
 -- get working on beagle.
 -- Debug Ideas:
     - Minimal C program which causes issues
@@ -24,17 +15,13 @@
     - Might be something that happens at link time:
         startup code for one of libraries
 
-VERY HIGH:
--- Benchmarks 
-    Timing: dramatic bencmark results for new proposal.
-    - Parallel bootstrap DONE
-    - Parallel CI
-    - smaller focused micro-tests looking at param size
+HIGH:
+-- Benchmark ideas
+    - Beagle
     - Sarah Kenny FMRI
-    - parallel OpenMx Tests
 
-HIGH:
--- Feedback from queue
+MED:
+-- Feedback from queue to let user know if job waiting in batch queue
 
 HIGH:
 -- better ctrl-c support
@@ -42,51 +29,32 @@
         cluster?
 
 HIGH:
--- Test on Mac
-
-HIGH:
 -- Usability testing package
 -- Instructions + survey
 
 MED:
- -- Plyr support
- -- foreach support
- -- raster support
+-- Support generic swift sites.xml and tc.data files for power users
 
+
 MED:
--- OpenMx specific instructions.
+-- automated tests
 
-HIGH:
--- Check  java version ahead of time
-
-HIGH:
--- email swift-devel about automated tests
-
 MED:
 -- IBI performance tests
 
 MED:
--- Add OpenMx tests to test suite
+-- Add OpenMx tests to SwiftR test suite
 
-HIGH:
-- perf approach
- -- socket?
- -- swift times?
- -- micro studies on provider staging etc.
-
 MID:
 - test on Ranger
 
 MID:
-- saner approach to channels: channel per request to avoid the issue
-    of what happens if a "done" is never read
+-- Check  java version ahead of time
 
 MID:
 - Support Swift mode where it runs multiple batch jobs to avoid timeout
     issues and congestion issues
 
-MID: 
-- mechanism to copy files across
  
 MID:
 - SGE for Tim bates
@@ -95,6 +63,8 @@
  -- siraf
  -- godzilla (if updated)
 
+MED:
+- sockets to improve performance?
 
 MID:
 * Cleanup ssh worker processes: add a watchdog that detects when worker.pl has 
@@ -105,12 +75,30 @@
 * restart batch jobs automatically if time runs out
 
 MID:
-Implement omxXXX parallel calls, update openmx code
+Implement omxXXX parallel calls that behave same on swift, snowfall, local:
+    * omxExport
+    * omxApply
+    * omxLibrary
+    * etc
+    * update openmx code
+    * add tests to openmx
 * Detect in OpenMx whether swift cluster is initialized.
 
+MED:
+
+Coaster timeout problem:
+    * Relevant bug: https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=253
+    * Temporary fix (increasing timeout duration) integrated into provider-coaster/src/org/globus/cog/abstraction/coaster/service/CoasterService.java
+    * Don't want to have to hardcode change:
+
 MID:
 - More robust mechanism for swiftExport - use swift to stage?
+- mechanism to copy other files across directly
 
+LOW:
+ -- Plyr support
+ -- foreach support
+ -- raster support
 LOW: (unless needed by immediate OpenMx app or test)
 - complete sf compat functions (sapply, lapply -> for openMx, based on usage)
 
@@ -147,6 +135,9 @@
 Investigate use of swift broadcase:
     swift bcast if exportAll files/data and/or xmit this via prov staging w/ caching of duplicates
 
+LOW:
+- saner approach to channels: channel per request to avoid the issue
+    of what happens if a "done" is never read
 
 =========================================================================
 Completed:
@@ -165,12 +156,8 @@
  Look at all OmxNNN parallel calls - see if any are used that we dont yet handle.
  - It turns out that they are not currently needed
 
-HIGH:
+        
 
-Coaster timeout problem:
-    He (Mihael) will also look at a better fix to the coaster timeout problem, but for now, you should integrate the timeout change from my trunk/cog/modules/provider-coaster/src/* into your test trunk/
-    Otherwise, you'll find that your coaster workers quit after a few minutes of inactivity and then start-swift needs to be killed, workers cleanup up, and start-swift restarted.
-
 MID:
 - user testing in general (ssh, pbs, sge) 
 * SGE testing: ranger, siraf (low priority)
@@ -181,17 +168,26 @@
  ssh server.  A convenience function getNodeList is included to make it easier
  to get the host list from the environment *)
 
-HIGH:
-See email about pbs breaking on UVa cluster
-
-
 MID:
-- Wiki: screen
-
-MID:
 - startup notes from Tim Bates
  -- rlib rpackage sugg   
 
-MID:
-* Fast branch: working? faster?
-    - DONE
+VERY HIGH:
+-- Streamlined tutorial for Multicore:
+    * Simple page with nothing aside from R commands for simplest case
+    * link to installation instructions
+    * link to advanced info
+
+VERY HIGH:
+-- OpenMx doco on wiki
+HIGH:
+-- Benchmark ideas
+    Timing: dramatic bencmark results for new proposal.
+    - Parallel bootstrap DONE
+    - Parallel CI
+    - smaller focused micro-tests looking at param size
+    - Sarah Kenny FMRI
+    - parallel OpenMx Tests
+
+HIGH:
+-- Test on Mac