[Swift-commit] r6330 - SwiftTutorials/IO

[davidk at ci.uchicago.edu]

Author: davidk
Date: 2013-03-05 10:29:26 -0600 (Tue, 05 Mar 2013)
New Revision: 6330

Added:
   SwiftTutorials/IO/TODO
Log:
TODO


Added: SwiftTutorials/IO/TODO
===================================================================
--- SwiftTutorials/IO/TODO	                        (rev 0)
+++ SwiftTutorials/IO/TODO	2013-03-05 16:29:26 UTC (rev 6330)
@@ -0,0 +1,32 @@
+lets desribe all the sources of file management and IO overhead that we know of, and how to eliminate them.
+1. strings to prevent staging
+2. use of scratch to reduce the effect of small-block IO (iff it applies: it might not apply here if the files are tiny)
+3. workdir overhead pushed to local node filesystem
+4. reduce/elim overhead of swift files like info and the .d dir
+
+You can do the same analysis/tests for CDM
+The trick there is ensuring that a) CDM files match as expected and (b) that CDM itself doesnt add unacceptable overhead
+ANd we have never stress tested the CDM file caching option that we always turn of in the cf file
+
+First one, just do a simplified test version of what you did for dssat on stampede?
+ie a version of that that strips out the "noise" and lists the things that made the improvement
+I think those things are the things I mentioned above
+SO, lets say for the app, you use a shell script that does 3 dd statements: the stagein dd, the "app" dd (using a small block size like 8KB) and the stageout dd ?
+one file in, ine file out, for starters.
+Later we can parameterize these tests and demonstrate the overhead of different aspects changing (ie add more in and/or out files) ?
+
+So this User Guide setion needs to start out very simple, but cover all the cases in a well-structured manner that makes it no more complex than is necessary.
+So I would structure this as a set of directories, where each dir has one test case/example and a README saying what it shows and how its used.
+You can just number the dirs, or give them longish names like "local_scratch+full_pathnames/"
+Each should have a cf and a sites.xml
+and save its workdirectory below it,
+and run an app that returns a detailed ls/find dummary of its cwd and arg list.
+because the added complexity is not just how the jobdir gets set, but also what gets passed to the app on its command line
+So, now, I *think* we have laidout out here all the issues:
+ - where is jobdir
+ - what is copied where
+ - what is linked where
+ - what is passed to command
+ - what is copied out after command
+ - what is cleaned up (under what options)
+