[Swift-user] Help needed with batching up parallel runs

[Ben Clifford]

Use clustering. Read the docs mike linked to. Basically you need to 
specify a maxwalltime for the jobs you want clustered, and then a 
clustering time that is some multiple (eg 10 in your case).

You might try coasters if you are submitting using GT2 (there is something 
wrong with gt4 + coasters at the moment that prevents them being used 
together).

On Mon, 21 Jul 2008, Tiberiu Stef-Praun wrote:

> Hi
> 
> I work with some code that generates at some point a number (300 in my
> case) of parallel identical runs, and I need to batch those up (10 at
> a time in my case) because each individual run is too short.
> I don't want Falkon at this point, and I'm not sure about the status
> of the coaster provider, so I would prefer a clean swift solution
> I was thinking of some array manipulation, but it was not obvious how
> to do it with swift.
> 
> Thanks !
> Tibi
> 
> Here is the code that I have so far, and I need help for:
> 
> 
> 
> //this is the code that batches a number of runs: based on the size of
> the array (determined where I make the call), I will return the set of
> parallel run results
> (file simFile[])gj_batch_sim(file policyFile,  file logFile){
>         app{
>                 gj_batch_sim @filename(policyFile) @filename(logFile)
> @filenames(simFile);
>         }
> }
> 
> int parallelInstances=300;
> file simOutputs[]<simple_mapper; prefix="sim_", suffix=".mat">;
> 
> (file simResults[])batch_gj_batch_sim(file policyFile, int parallelInstances){
>         // this is just some needed input
>         file logFile<single_file_mapper; file="inputs/gj_log76.mat">;
> 
>        // I want to have batches of size 10
>         int localBatchSize=10;
> 
>         int batchRange=@toint(@strcut(@strcat(parallelInstances/localBatchSize),"([0-9]*).?[0-9]*")
>         trace("Times to do batch_gj_batch_sim",batchRange);
> 
>         foreach i in  [1:batchRange] {
>                 // HELP HERE: how to do this ?
>                 // essentially I need to map the proper batch of file
> names into the call of gj_batch_sim
> 
> simResults[batchSize*i:batchSize*(i+1)-1]=gj_batch_sim(policyFile,
> logFile);
>         }
> }
> 
> 
> 
>