[Swift-commit] r5524 - SwiftApps/SciColSim
wilde at ci.uchicago.edu
wilde at ci.uchicago.edu
Sun Jan 29 19:47:21 CST 2012
Author: wilde
Date: 2012-01-29 19:47:21 -0600 (Sun, 29 Jan 2012)
New Revision: 5524
Added:
SwiftApps/SciColSim/test-orig.sh
SwiftApps/SciColSim/test-swift.sh
SwiftApps/SciColSim/testopt.py
Removed:
SwiftApps/SciColSim/testO.sh
SwiftApps/SciColSim/testopt1.py
Modified:
SwiftApps/SciColSim/EMAIL
SwiftApps/SciColSim/README
SwiftApps/SciColSim/TODO
SwiftApps/SciColSim/annealing.swift
SwiftApps/SciColSim/evolve.sh
SwiftApps/SciColSim/local.xml
SwiftApps/SciColSim/optimizer.cpp
SwiftApps/SciColSim/sumloss.sh
Log:
Checkpoint of doc and code devel.
Modified: SwiftApps/SciColSim/EMAIL
===================================================================
--- SwiftApps/SciColSim/EMAIL 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/EMAIL 2012-01-30 01:47:21 UTC (rev 5524)
@@ -207,7 +207,7 @@
> 15 repeats (parallel)
> 100 Annealing_cycles (serial)
> 6 repeats (serial)
-> 1000 to 10000 annealing_repeats (parallel)
+> 1000 to 10000 annealing_repeats (parallel) # NOTE (1/29 mw): should this be called "reruns"?
> evolve()
>
> The main question I have at this point is regarding the strategy for
Modified: SwiftApps/SciColSim/README
===================================================================
--- SwiftApps/SciColSim/README 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/README 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,38 +1,102 @@
-=== Overview
+*** Overview
The code simulates the exploration of a chemical network by a research
community. The five major parameters determine the strategy of
exploration, "target" is the number of new interactions discovered and
"loss" is the number of experiments per one new positive finding.
-=== Files
+Terminology and application logic:
+ optimizer: the C++ based optimizer that performs simulated annealing
+ optimizetion of parameters to the evolve() function.
+
+ evolve(): the actual science code which does the "collaboration
+ simulation" desribed in Overview above. Implemented as a
+ function within optimizer. Uses the boost graph lib.
+ Calculates "loss" - amount of effort expended in rerunning
+ scientific experiments.
+
+ multi_annealing(): the function within optimizer that does the
+ simulated annealing optimization logic (multiple times)
+
+ multi_loss: performs a total of N independent repetitions of evolve()
+ on NWorker cores, in parallel. Each evolve() returns one loss
+ value, a scalar double. Returns average loss over all evolve()
+ simulations, and std dev.
+
+*** Files
+
Code:
+testopt.py: top-most script to run the C++ optimizer (with no
+ swift). Based on a simpler Python script provided by
+ Andrey to run the C++ optimizer.
+
+test-orig.sh: original python script test script for optimizer,
+ provided by Andrey with some mods by Mike.
+
optirun.swift: replaces top-level py script for the outermost loops
+ (temporarily deprecated: has not been maintained to match changes in optimizer)
+annealing.swift: the annealing logic of optimizer, moved to
+ swift. Calls optimizer(op=m) to run 1 multi_loss() of N
+ evolve() runs on NWorker cores. Implements a higher-level multi_loss
+ which spreads the N reruns of evolve() between M runs of optimizer(m)
+ each of which does R evolve runs. N = P*R. Eg, n=2400, P=100, R=24,
+ would use 100 nodes (invocations) of 24 cores each.
+
+
+tc specifies:
+
+ optimizer.sh (old) / evolve.sh (current)
+
+ sumloss.sh: calculates avg loss and its std dev
+ did this as a separate app because it was hard to do the sum()
+ logic in swift. How best to do that? Try recursion? fold()?
+
+
Data:
+ input: movie_data.txt
+
+ output: best_??? max? itermiediate? FIX
-=== How to build
-make
+*** How to build
-=== How to Run
+Just run "make" -> uses Makefile
-./test0.sh
+ on mac, makes: openmp-optimizer (multi_loss() changed to use OpenMP),
+ dispatch-optimizer (orig code logic, only slight corrections)
-./testO.sh
+ on linux, makes: openmp-optimizer
-./test1.sh
-./*.py
+*** How to Run
+Non-swift tests:
-=== C++ app flow logic ===
+ cp testopt.py t1.py etc; then edit t1.py etc to set the params you want.
-for target in range(58, 1009 (used 209), 50): // 20 values
- for i in range(15):
+Swift tests:
+
+ cp testswift.sh t1.sh etc; then edit t1.sh etc to set the params you want.
+
+
+In each of the above, set the optimizer version you want to run.
+
+
+*** C++ Command line args
+
+
+*** Swift command line args and internal control vars
+
+
+
+*** C++ app flow logic ===
+
+for target in range(58, 1009 (used 209), 50): // 20 values Target Values
+ for i in range(15): Optimization repeats (runs?)
# P0: P1: D: P2: P3:
optimizer |0 0 4 50 -1 target | 40000 20 1000 2 | 1 | 2. 0.01 100 0.3 2.3 | 1 1 0 0 0
@@ -44,7 +108,7 @@
curr_err = Res.second;
for i = 0 to annealing_cycles - 1 (100 from .py script)
- for j = 0 to 4 // 5X: 0..4 is fixed constant
+ for j = 0 to 4 // 5X: 0..4 is fixed constant - one iteration for each of 5 evolve() parameters
setParameters (for a round of parallel Annealing_repeats
Res = multi_loss(un, Results, Counters, Annealing_repeats);
In parallel: for 1 to AnnealingRepeats (1,000; 10,000 desired)
@@ -52,14 +116,20 @@
evolve()
setParameters again here, conditionally? (check)
-20 targets - py
- 15 repeats - py
- 1 initial multi_loss: 1000 to 10000 annealing_repeats
- 100 Annealing_cycles (groups of 10? : cycle=10 ) (fast:50)
- 5 repeats (fast: 1)
- multi_loss: 1000 to 10000 evolve re-runs
- evolve() => 2 mins to 10 mins
+TODO: Merge these two outlines into one:
-=== END
+P 20 targets - py (target loss)
+P 15 repeats - py
+S 1 initial multi_loss: 1000 to 10000 annealing_repeats
+S 100 Annealing_cycles (groups of 10? : cycle=10 ) (fast:50)
+S 5 repeats (fast: 1)
+P multi_loss: 1000 to 10000 evolve re-runs
+S evolve() (leaf) => 2 mins to 10 mins (is that time for one or for 1000?)
-
+# a) 20 targets (parallel)
+# b) 15 repeats (parallel) of optimizer (optimizer == multi_annealing):
+# 1 multi_loss to initialize
+# c) 100 Annealing_cycles (serial)
+# d) 5 repeats (1 per param, serial) of multi_loss:
+# e) 1000 to 10000 annealing_repeats (parallel) == multi_loss
+# f) evolve()
Modified: SwiftApps/SciColSim/TODO
===================================================================
--- SwiftApps/SciColSim/TODO 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/TODO 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,37 +1,69 @@
-[x] Determine if one evolve() run in the optimizer.cpp code is
-sufficient. (yes -it does 1 woth reruns = 1
+Stopped at:
+Are deps working right? why 20 in args? (not 10?) Why 2 calls in progress? Whats with initial call [0][0]?
+
+trace nan error; verify if N reruns per call is working and sumloss handles it OK.
+
+---
+
+x = done, - = in progress
+
[ ] Determine correct test parameter sets - and if we are passing these
-through correctly.
+ through correctly.
-[ ] Check correctnes of rejection tracking logic and j of k logic for the 5 params
+[ ] Check correctnes of rejection tracking logic and j of k logic for
+ the 5 params
[ ] address fixme's
-[ ] deterine how to find max_reject
+[ ] determine how to find max_reject
[ ] Determine if and how to let the optimizer.cpp code do N reruns and
-how to set NWorkers accordingly (must evenly divide in current code).
-Check if floor() is correct and how load is partitioned; what N makes
-sense in swift usage?
+ how to set NWorkers accordingly (must evenly divide in current
+ code). Check if floor() is correct and how load is partitioned; what
+ N makes sense in swift usage?
-[x] Determine annealing.swift output files needed.
+[x] Ensure we are doing 100's to 1000's of evolve() calls in parallel.
-[_x_ ] Make annealing.swift stdout/err output match that of optimizer.cpp
+[ ] Do performance tests on Beagle and then other platforms (OSG, TG)
+ - at scale.
-[x] Integrate code to start with usec seed
+[-] Make annealing.swift stdout/err output match that of optimizer.cpp
-[x] add code to set secific seed for reproducibiity. Will Swift use same from Java Random?
-
[ ] Align Swift and .cp code enough do a correctness test.
[ ] Enable precision control for %f formatting (to match c++
-output). Why are we getting some long and some short? Is swift
-truncating the constant or the tracef output?
+ output). Why are we getting some long and some short? Is swift
+ truncating the constant or the tracef output?
-[ ] Do performance tests on Beagle and then other platforms (OSG, TG)
-- at scale. Ensure we aredoing 100's to 1000's of evolve() calls in
-parallel.
-
[ ] Package code for ExM research.
+[ ] add logic from C++ called "random intiation of starting parameters
+
+[ ] make calls to optimizer(op m) work right regardless of what N and
+ Nworkers is. Now seems to require ((N mod NWorkers0 == 0)
+
+[-] Platforms:
+
+[-] Mac
+[ ] Beagle
+[ ] OSG
+[ ] XSEDE
+[ ] OSG+XSEDE
+[ ] BG/P
+[ ] PADS
+[ ] FutureGrid
+[ ] OSG Cloud
+[ ] ibiCluster
+
+---
+
+[x] Integrate code to start with usec seed
+
+[x] add code to set secific seed for reproducibiity. Will Swift use
+same from Java Random?
+
+[x] Determine if one evolve() run in the optimizer.cpp code is
+sufficient. (yes -it does 1 woth reruns = 1
+
+[x] Determine annealing.swift output files needed.
Modified: SwiftApps/SciColSim/annealing.swift
===================================================================
--- SwiftApps/SciColSim/annealing.swift 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/annealing.swift 2012-01-30 01:47:21 UTC (rev 5524)
@@ -10,7 +10,7 @@
global boolean FIX_VARIABLES = true;
global int var_fixed[] = [1,1,0,0,0];
-global int Nworkers = 1;
+global int Nworkers = 4;
(float nx) newx(float x, float dx)
{
@@ -63,7 +63,7 @@
float x[][], dx[][], curr_loss[], curr_sdev[];
Res mlres[][];
- mlres[0][0] = multi_loss( 0, 0, params0, target_innov, evolve_reruns ); // Only done once, not 5x; serves for all evolve-params ???
+ mlres[0][0] = multi_loss( 0, 0, params0, target_innov, evolve_reruns ); // FIXME: serves for all evolve-params ???
tracef("multi_annealing: AR: %f +- %f\n",mlres[0][0].loss,mlres[0][0].sdev);
foreach j in [0:NEVOPARAMS-1] {
@@ -77,7 +77,7 @@
foreach i in [1:annealing_cycles] {
// set new temperature, rejection threshold, and dx values for this cycle
float temperature = T_start*exp( @tofloat(i)*(jlog(T_end)-jlog(T_start))/@tofloat(annealing_cycles));
- tracef("in multi_annealing: AR: i=%i ....T=%f\n", i, temperature);
+ // tracef("in multi_annealing: AR: i=%i ....T=%f\n", i, temperature);
// On each new "major" cycle within the annealing_cycles (other than the first) set new rejection and dx values
if ( i %% cycle == 1 && i > 1 ){
tracef("multi_annealing: new cycle at i=%i\n",i);
@@ -93,7 +93,7 @@
}
trace ("Recomputed rejection: i=%d k=%d dx[i][k]=%f\n", i, k, dx[i][k]);
}
- tracef("in multi_annealing: AR: %f %f %f %f %f\n",dx[i][0],dx[i][1],dx[i][2],dx[i][3],dx[i][4],dx[i][5]);
+ tracef("multi_annealing: AR: %f %f %f %f %f\n",dx[i][0],dx[i][1],dx[i][2],dx[i][3],dx[i][4],dx[i][5]);
}
else { # If not new cycle, set dx from previous dx (i-1)
foreach k in [0:NEVOPARAMS-1] {
@@ -104,7 +104,7 @@
// float try_x[];
int curr = (i * NEVOPARAMS) + j;
int prev = curr-1;
- tracef("in multi_annealing: i=%i j=%i curr=%i prev=%i\n", i, j, curr, prev);
+ // tracef("in multi_annealing: i=%i j=%i curr=%i prev=%i\n", i, j, curr, prev);
if ( /*(!FIX_VARIABLES) || */ (var_fixed[j]==0) ) {
# fixed=1,1,0,0,0: FIXME: FIX_VARIABLES flag has faulty logic but OK when TRUE
float try_x[];
@@ -120,15 +120,15 @@
try_x[k] = x[i-1][k]; // use x[i-1][k] (from prior cycle)
}
}
- tracef("in multi_annealing: i=%i j=%i k=%i try_x=%f\n", i, j, k, try_x[k]);
- tracef("in multi_annealing: i=%i j=%i k=%i\n", i, j, k);
+ // tracef("in multi_annealing: i=%i j=%i k=%i try_x=%f\n", i, j, k, try_x[k]);
+ // tracef("in multi_annealing: i=%i j=%i k=%i\n", i, j, k);
}
// Up to here, x[] and dx[] are only set for previous i
- tracef( @strcat(color(3,"in multi_annealing: "),"i=%i j=%i calling multi_loss\n"),i,j);
- #tracef("multi_annealing: i=%i j=%i calling multi_loss try_x=%q\n",i,j,try_x);
+ # tracef( @strcat(color(3,"in multi_annealing: "),"i=%i j=%i calling multi_loss\n"),i,j);
+ # tracef("multi_annealing: i=%i j=%i calling multi_loss try_x=%q\n",i,j,try_x);
mlres[i][j] = multi_loss(i,j,try_x, target_innov, evolve_reruns); # do the N evolve()'s, N=evolve_reruns
- tracef("multi_annealing: i=%i j=%i ret vals from multi_loss=(%f,%f)\n",i,j,mlres[i][j].loss,mlres[i][j].sdev);
- tracef("in multi_annealing: AR: %f+-%f\n",mlres[i][j].loss,mlres[i][j].sdev);
+ # tracef("multi_annealing: i=%i j=%i ret vals from multi_loss=(%f,%f)\n",i,j,mlres[i][j].loss,mlres[i][j].sdev);
+ # tracef("in multi_annealing: AR: %f+-%f\n",mlres[i][j].loss,mlres[i][j].sdev);
// Beyond this point, x[] and dx[] are being set for this i,j
float ratio = min(1.0, exp( -(mlres[i][j].loss-curr_loss[prev]) /temperature));
float r = (random()) / (pow(2.0,31.0)-1.0); // why all the 2^31's ???
@@ -199,7 +199,11 @@
file rfile[];
tracef("multi_loss: entered: ci=%i cj=%i target_innov=%f evolve_reruns=%i x=%q\n",ci, cj, target_innov,evolve_reruns,x);
- foreach i in [1:evolve_reruns] { // repeats of the evolove() - same as n_reruns
+ int rerunsPerApp = 120;
+ int appCalls = @toint(@tofloat(evolve_reruns) / @tofloat(rerunsPerApp)); # FIXME: handle fractional issues and rounding etc. For now must divide evenly
+
+tracef("multi_loass appCalls=%i\n", appCalls);
+ foreach i in [1:appCalls] { // repeats of the evolove() - same as n_reruns
file outfile; // FIXME: map and save in future
string args[] = [ // FIXME: move this to a setargs() function
// alpha_i alpha_m beta gamma delta target_innov
@@ -207,7 +211,7 @@
// n_epochs n_steps evolve_reruns range
// "40000", "20", @strcat(evolve_reruns), "2",
- "40000", "20", "1", "2", # Set reruns to 1 for the cpp code; we do the re-runs in Swift for now.
+ "40000", "20", @strcat(rerunsPerApp), "2",
// verbose_level
"1",
@@ -218,12 +222,13 @@
// FREEZE: alpha_i alpha_m beta gamma delta
"1", "1", "0", "0", "0",
- // operation-code:(m,a) Nworkers
- "m", @strcat(Nworkers) ];
+ // operation-code:(m,a) Nworkers seed
+ "m", @strcat(Nworkers), "1234567" ];
file graph <"movie_graph.txt">;
(outfile, rfile[i]) = evolve(args,graph);
- tracef("multi_loss: after evolve: i=%i %k %k\n", i, outfile, rfile[i]);
+ tracef("multi_loss: i=%i calling evolve, args=%q\n", i, args);
+ // tracef("multi_loss: after evolve: i=%i %k %k\n", i, outfile, rfile[i]);
}
file sumfile = sumloss(rfile);
r = readData(sumfile);
@@ -233,7 +238,7 @@
optimizer_sweep() // Implements logic of python driver script
{
int minrange=58;
- int maxrange=209;
+ int maxrange=59;
//int maxrange=59;
//int maxrange=1009;
//int maxrange=209;
@@ -241,8 +246,7 @@
// fixme: add provision for random priming and random param values when x[i] == -100 (see optimizer.cpp main())
- //int nreps=1; # 15
- int nreps=3; # 15
+ int nreps=1; # 15
// file bestfile <single_file_mapper; file=@strcat("output/T",target,".R",rep,".best_opt_some")>;
// file maxfile <single_file_mapper; file=@strcat("output/T",target,".R",rep,".max_dist")>;
@@ -266,11 +270,11 @@
2.0,
0.01,
0.3,
- 10,
+ 1200,
2.3,
[0.0, 0.0, 4.0, 50.0, -1.0],
@tofloat(target_innov),
- 1);
+ 5);
}
}
}
Modified: SwiftApps/SciColSim/evolve.sh
===================================================================
--- SwiftApps/SciColSim/evolve.sh 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/evolve.sh 2012-01-30 01:47:21 UTC (rev 5524)
@@ -2,5 +2,6 @@
datafile=$1
touch multi_loss.data
shift 1
-$(dirname $0)/openmp-optimizer $* 2>&1
+NWORKERS=$23
+OMP_NUM_THREADS=$NWORKERS $(dirname $0)/openmp-optimizer $* 2>&1
mv multi_loss.data $datafile
Modified: SwiftApps/SciColSim/local.xml
===================================================================
--- SwiftApps/SciColSim/local.xml 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/local.xml 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,7 +1,7 @@
<config>
<pool handle="localhost" >
<execution provider="local" url="none" />
- <profile namespace="karajan" key="jobThrottle">0.23</profile>
+ <profile namespace="karajan" key="jobThrottle">0.00</profile>
<profile namespace="karajan" key="initialScore">10000</profile>
<filesystem provider="local"/>
<workdirectory>/tmp/wilde/swiftwork</workdirectory>
Modified: SwiftApps/SciColSim/optimizer.cpp
===================================================================
--- SwiftApps/SciColSim/optimizer.cpp 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/optimizer.cpp 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1402,6 +1402,7 @@
#else
Res = multi_loss( un, Results, Counters, x);
#endif
+ std::cout << "Returned from initial multi_loss:" << std::endl;
std::cout << Res.first << " +- " << Res.second << std::endl;
if ( operation == 'm' ) {
Modified: SwiftApps/SciColSim/sumloss.sh
===================================================================
--- SwiftApps/SciColSim/sumloss.sh 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/sumloss.sh 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,17 +1,25 @@
#! /bin/awk -f
-BEGIN { n = ARGC-1 }
+BEGIN { n = 0; loss = 0; }
+
{
- loss += $1/n
- loss_sq += ($1*$1)/n
+ loss += $1
+ loss_sq += ($1*$1)
+ n++
}
+
END {
+
+ loss /= n;
+ loss_sq /= n;
+
x = (loss_sq - (loss*loss))/n
sdev = 2.0 * sqrt(x)
printf "loss sdev\n"
printf "%f %f\n", loss, sdev
}
+
# the awk script above implements this c++ logic from optimizer.cpp:
#
# for (int i=0; i<N; i++){
@@ -29,3 +37,18 @@
# std::pair<double,double> Res;
# Res.first=Loss;
# Res.second=two_std;
+
+
+# Older logic, literal transcription from C++ (didnt handle multiline files correctly):
+
+# {
+# loss += $1/n
+# loss_sq += ($1*$1)/n
+# }
+#
+# {
+# x = (loss_sq - (loss*loss))/n
+# sdev = 2.0 * sqrt(x)
+# printf "loss sdev\n"
+# printf "%f %f\n", loss, sdev
+# }
Copied: SwiftApps/SciColSim/test-orig.sh (from rev 5519, SwiftApps/SciColSim/testO.sh)
===================================================================
--- SwiftApps/SciColSim/test-orig.sh (rev 0)
+++ SwiftApps/SciColSim/test-orig.sh 2012-01-30 01:47:21 UTC (rev 5524)
@@ -0,0 +1,21 @@
+#! /bin/bash
+
+python <<END
+import os
+
+target = 58
+s = ("%d" % target)
+
+args="0 0 4 50 -1 "+s+" 40000 20 1000 2 1 2. 0.01 100 0.3 2.3 1 1 0 0 0 "
+args="0 0 4 50 -1 "+s+" 40000 20 100 2 1 2. 0.01 2 0.3 2.3 1 1 0 0 0 "
+
+print("\n\n **** CALLING APP: "+args+"\n\n\n")
+os.system("./optimizer "+args+" >out.o.T"+s);
+os.system("./Optimizer "+args+" >out.O.T"+s);
+
+# print("\n\n **** CALLING APP: ./optimizer 0 0 4 50 -1 "+s+" 40000 20 10 2 1 2. 0.01 100 0.3 2.3 1 1 0 0 0\n\n\n")
+# FAST: os.system("./optimizer 0 0 4 50 -1 "+s+" 40000 20 10 2 1 2. 0.01 2 0.3 2.3 1 1 0 0 0")
+
+print "Done!"
+END
+
Added: SwiftApps/SciColSim/test-swift.sh
===================================================================
--- SwiftApps/SciColSim/test-swift.sh (rev 0)
+++ SwiftApps/SciColSim/test-swift.sh 2012-01-30 01:47:21 UTC (rev 5524)
@@ -0,0 +1,7 @@
+escapecode=$(echo -n -e '\033')
+
+swift -config cf -tc.file tc -sites.file local.xml annealing.swift \
+ -e33="$escapecode"
+
+# -evolveReruns=20 \
+# -annealingCycles=10
Property changes on: SwiftApps/SciColSim/test-swift.sh
___________________________________________________________________
Added: svn:executable
+ *
Deleted: SwiftApps/SciColSim/testO.sh
===================================================================
--- SwiftApps/SciColSim/testO.sh 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/testO.sh 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,21 +0,0 @@
-#! /bin/bash
-
-python <<END
-import os
-
-target = 58
-s = ("%d" % target)
-
-args="0 0 4 50 -1 "+s+" 40000 20 1000 2 1 2. 0.01 100 0.3 2.3 1 1 0 0 0 "
-args="0 0 4 50 -1 "+s+" 40000 20 100 2 1 2. 0.01 2 0.3 2.3 1 1 0 0 0 "
-
-print("\n\n **** CALLING APP: "+args+"\n\n\n")
-os.system("./optimizer "+args+" >out.o.T"+s);
-os.system("./Optimizer "+args+" >out.O.T"+s);
-
-# print("\n\n **** CALLING APP: ./optimizer 0 0 4 50 -1 "+s+" 40000 20 10 2 1 2. 0.01 100 0.3 2.3 1 1 0 0 0\n\n\n")
-# FAST: os.system("./optimizer 0 0 4 50 -1 "+s+" 40000 20 10 2 1 2. 0.01 2 0.3 2.3 1 1 0 0 0")
-
-print "Done!"
-END
-
Copied: SwiftApps/SciColSim/testopt.py (from rev 5522, SwiftApps/SciColSim/testopt1.py)
===================================================================
--- SwiftApps/SciColSim/testopt.py (rev 0)
+++ SwiftApps/SciColSim/testopt.py 2012-01-30 01:47:21 UTC (rev 5524)
@@ -0,0 +1,144 @@
+#! /usr/bin/env python
+
+import os, sys
+
+unbuffered = os.fdopen(sys.stdout.fileno(), 'w', 0)
+sys.stdout = unbuffered
+
+app = "./orig-optimizer"; # For Mac only: original code (+1 loop fix) using Grand Central Dispatch
+app = "./dispatch-optimizer"; # For Mac only: sing Grand Central Dispatch
+app = "./openmp-optimizer"; # For Mac or Linux: Using OpenMP (Default)
+
+paramset="default"
+
+if paramset == "default":
+
+ # Parameter names and optimizer command line positions
+
+ startTarget = 58 # target (target innovation)
+ endTarget = 1009 # "
+ incrTarget = 50 # "
+ optimizerRepeats = 15 # wrapper control var (# of repeats of optimizer to do for each target loss)
+ evolveReruns = 1000 # n_reruns =
+ annealingSteps = 100
+
+# Following params are for modified optimizer openmp-optimizer
+
+# NWorkers = "2"
+# operation = "n" # n=normal, m=manual (runs 1 multi_loss call)
+# seed = "1234567"
+# openmp = "OMP_NUM_THREADS=24"
+
+ NWorkers = ""
+ operation = ""
+ seed = ""
+ openmp = ""
+
+elif paramset == "FULLSCALE": # FULL INITIAL TEST PARAMETERS from Andrey
+
+ startTarget = 58
+ endTarget = 1009
+ incrTarget = 50
+ optimizerRepeats = 15
+ evolveReruns = 1000
+ annealingSteps = 100
+
+elif paramset == "FASTTEST": # FAST TEST PARAMETERS by Mike
+
+ startTarget = 58
+ endTarget = 209
+ incrTarget = 50
+ optimizerRepeats = 2
+ evolveReruns = 10
+ annealingSteps = 2
+ NWorkers = "2"
+ openmp = "OMP_NUM_THREADS=" + NWorkers
+ operation = "n" # n=normal, m=manual (runs 1 multi_loss call)
+ seed = "1234567"
+
+elif paramset == "MANUALTEST": # for manual (one multi_loss call) mode
+
+ startTarget = 258 # runtime increases with greater target
+ endTarget = 259
+ incrTarget = 50
+ optimizerRepeats = 1
+ evolveReruns = 60
+ annealingSteps = 5
+ NWorkers = "4"
+ openmp = "OMP_NUM_THREADS=" + NWorkers
+ operation = "m" # n=normal, m=manual (runs 1 multi_loss call)
+ seed = "1234567"
+
+# Ensure we dont pass new parameters to original optimizer versions
+# (but they would be ignored)
+
+if app == "./orig-optimizer":
+ NWorkers = ""
+ operation = ""
+ seed = ""
+
+# Run the test
+
+for target in range(startTarget,endTarget,incrTarget):
+ for i in range(optimizerRepeats):
+ args = openmp + " " + app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \
+ " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 1 1 0 0 0 " + operation + " " + NWorkers + " " + seed;
+ print("\n**** Calling optimizer: "+args+"\n")
+ os.system(args);
+
+print sys.argv[0] + " Done!"
+
+
+### Argument worksheet:
+
+# string fastargs1[] = [
+# "0", "0", "4", "50", "-1", @strcat(target),
+# "40000", "20", "1000", "2",
+# "1",
+# "2.", "0.01", "100", "0.3", "2.3",
+# "1", "1", "0", "0", "0"];
+# string fastargs2[] = [
+# "0", "0", "4", "50", "-1", @strcat(target),
+# "40000", "20", "1000", "2",
+# "1",
+# "2.", "0.01", "5", "0.3", "2.3",
+# "1", "1", "0", "0", "0", "m"];
+# string fastargs3[] = [
+# "0", "0", "4", "50", "-1", @strcat(target),
+# "40000", "20", @strcat(repeats), "2",
+# "1",
+# "2.", "0.01", "2", "0.3", "2.3",
+# "1", "1", "0", "0", "0", "m"];
+#
+#
+# long running args = "0 0 4 50 -1 target | 40000 20 1000 | 2 | 1 | 2. 0.01 100 0.3 2.3 | 1 1 0 0 0 | m 24? 12345
+#
+# [ alpha_i alpha_m beta gamma delta target_innov ]
+# 0 0 4 50 -1 target
+#
+# [ n_epochs n_steps n_reruns ] [ range ] [ verbose_level]
+# 40000 20 1000 2 1
+#
+# [ T_start T_end Annealing_steps Target_rejection Starting_jump ]
+# 2. 0.01 100 0.3 2.3
+#
+# [ FREEZE_alpha_i FREEZE_alpha_m FREEZE_beta FREEZE_gamma FREEZE_delta ]
+# 1 1 0 0 0
+#
+# [ operation-code:(n,m) Nworkers seed ]
+# n 24 12345
+#
+
+# From older test, using separate file per run:
+# args = app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \
+# " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 1 1 0 0 0 > out.T"+str(target)+".i"+str(i)
+
+### Optimization loop logic:
+
+# a) 20 targets (parallel)
+# b) 15 repeats (parallel) of optimizer (optimizer == multi_annealing):
+# 1 multi_loss to initialize
+# c) 100 Annealing_cycles (serial)
+# d) 5 repeats (1 per param, serial) of multi_loss:
+# e) 1000 to 10000 annealing_repeats (parallel) == multi_loss
+# f) evolve()
Deleted: SwiftApps/SciColSim/testopt1.py
===================================================================
--- SwiftApps/SciColSim/testopt1.py 2012-01-29 04:43:37 UTC (rev 5523)
+++ SwiftApps/SciColSim/testopt1.py 2012-01-30 01:47:21 UTC (rev 5524)
@@ -1,61 +0,0 @@
-#! /usr/bin/env python
-# -u => run unbuffered
-
-
-
-import os, sys
-
-
-#_-----------
-
-unbuffered = os.fdopen(sys.stdout.fileno(), 'w', 0)
-sys.stdout = unbuffered
-
-#--------------
-
-
-app = "./orig-optimizer"; # For Mac only: original code (+1 loop fix) using Grnd Central Dispatch
-app = "./dispatch-optimizer"; # For Mac only: sing Grand Central Dispatch
-app = "./openmp-optimizer"; # For Mac or Linux: Using OpenMP (Default)
-
-app = "./openmp-optimizer";
-
-# FULL INITIAL TEST PARAMETERS from Andrey
-
-startTarget = 58
-endTarget = 1009
-incrTarget = 50
-optimizerRepeats = 15
-evolveReruns = 1000
-annealingSteps = 100
-
-# FAST TEST PARAMETERS by Mike
-
-startTarget = 58
-endTarget = 209
-incrTarget = 50
-optimizerRepeats = 2
-evolveReruns = 10
-annealingSteps = 2
-
-# extra (string) parameters for openmp- and dispatch- versions only:
-
-NWorkers = "2"
-operation = "n" # n=normal, m=manual (runs 1 multi_loss call)
-seed = "1234567";
-
-if app == "./orig-optimizer":
- NWorkers = ""
- operation = ""
-
-for target in range(startTarget,endTarget,incrTarget):
- for i in range(optimizerRepeats):
- args = app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \
- " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 1 1 0 0 0 " + operation + " " + NWorkers + " " + seed;
- print("\n**** Calling optimizer: "+args+"\n")
- os.system(args);
-print sys.argv[0] + " Done!"
-
-# From older test, using separate file per run:
-# args = app + " 0 0 4 50 -1 " + str(target) + " 40000 20 " + str(evolveReruns) + \
-# " 2 1 2. 0.01 " + str(annealingSteps) + " 0.3 2.3 1 1 0 0 0 > out.T"+str(target)+".i"+str(i)
More information about the Swift-commit
mailing list