[Swift-user] Help needed with batching up parallel runs

[Michael Wilde]

Tibi, can you use the Swift clustering mechanism?
http://www.ci.uchicago.edu/swift/guides/userguide.php#clustering
Its meant for this sort of thing, and is nice because you dont need to 
explicitly do the clustering in your Swift script.

"Swift can group a number of short job submissions into a single larger 
job submission to minimize overhead involved in launching jobs..."

- Mike


On 7/21/08 10:39 AM, Tiberiu Stef-Praun wrote:
> Hi
> 
> I work with some code that generates at some point a number (300 in my
> case) of parallel identical runs, and I need to batch those up (10 at
> a time in my case) because each individual run is too short.
> I don't want Falkon at this point, and I'm not sure about the status
> of the coaster provider, so I would prefer a clean swift solution
> I was thinking of some array manipulation, but it was not obvious how
> to do it with swift.
> 
> Thanks !
> Tibi
> 
> Here is the code that I have so far, and I need help for:
> 
> 
> 
> //this is the code that batches a number of runs: based on the size of
> the array (determined where I make the call), I will return the set of
> parallel run results
> (file simFile[])gj_batch_sim(file policyFile,  file logFile){
>         app{
>                 gj_batch_sim @filename(policyFile) @filename(logFile)
> @filenames(simFile);
>         }
> }
> 
> int parallelInstances=300;
> file simOutputs[]<simple_mapper; prefix="sim_", suffix=".mat">;
> 
> (file simResults[])batch_gj_batch_sim(file policyFile, int parallelInstances){
>         // this is just some needed input
>         file logFile<single_file_mapper; file="inputs/gj_log76.mat">;
> 
>        // I want to have batches of size 10
>         int localBatchSize=10;
> 
>         int batchRange=@toint(@strcut(@strcat(parallelInstances/localBatchSize),"([0-9]*).?[0-9]*")
>         trace("Times to do batch_gj_batch_sim",batchRange);
> 
>         foreach i in  [1:batchRange] {
>                 // HELP HERE: how to do this ?
>                 // essentially I need to map the proper batch of file
> names into the call of gj_batch_sim
> 
> simResults[batchSize*i:batchSize*(i+1)-1]=gj_batch_sim(policyFile,
> logFile);
>         }
> }
> 
> 
>