[Swift-user] Help needed with batching up parallel runs

[Ben Clifford]

If you're using GRAM4 to submit, then it looks like you are hitting a bug 
that I fixed a week or so ago, cog svn r2066, which deals with the way 
that walltimes are formatted.

On Wed, 23 Jul 2008, Tiberiu Stef-Praun wrote:

> Hi, thanks, I forgot about that.
> I tried running it on teraport, and it failed.
> The log is here:
> http://www.ci.uchicago.edu/~tiberius/issues/gj-batched-20080723-1522-fypk29g6.log
> 
> Mihael suggested that I should file a bug report, but I'm not sure
> what to report (other than the failure)
> 
> Tibi
> 
> On Tue, Jul 22, 2008 at 2:14 AM, Ben Clifford <benc at hawaga.org.uk> wrote:
> >
> > Use clustering. Read the docs mike linked to. Basically you need to
> > specify a maxwalltime for the jobs you want clustered, and then a
> > clustering time that is some multiple (eg 10 in your case).
> >
> > You might try coasters if you are submitting using GT2 (there is something
> > wrong with gt4 + coasters at the moment that prevents them being used
> > together).
> >
> > On Mon, 21 Jul 2008, Tiberiu Stef-Praun wrote:
> >
> >> Hi
> >>
> >> I work with some code that generates at some point a number (300 in my
> >> case) of parallel identical runs, and I need to batch those up (10 at
> >> a time in my case) because each individual run is too short.
> >> I don't want Falkon at this point, and I'm not sure about the status
> >> of the coaster provider, so I would prefer a clean swift solution
> >> I was thinking of some array manipulation, but it was not obvious how
> >> to do it with swift.
> >>
> >> Thanks !
> >> Tibi
> >>
> >> Here is the code that I have so far, and I need help for:
> >>
> >>
> >>
> >> //this is the code that batches a number of runs: based on the size of
> >> the array (determined where I make the call), I will return the set of
> >> parallel run results
> >> (file simFile[])gj_batch_sim(file policyFile,  file logFile){
> >>         app{
> >>                 gj_batch_sim @filename(policyFile) @filename(logFile)
> >> @filenames(simFile);
> >>         }
> >> }
> >>
> >> int parallelInstances=300;
> >> file simOutputs[]<simple_mapper; prefix="sim_", suffix=".mat">;
> >>
> >> (file simResults[])batch_gj_batch_sim(file policyFile, int parallelInstances){
> >>         // this is just some needed input
> >>         file logFile<single_file_mapper; file="inputs/gj_log76.mat">;
> >>
> >>        // I want to have batches of size 10
> >>         int localBatchSize=10;
> >>
> >>         int batchRange=@toint(@strcut(@strcat(parallelInstances/localBatchSize),"([0-9]*).?[0-9]*")
> >>         trace("Times to do batch_gj_batch_sim",batchRange);
> >>
> >>         foreach i in  [1:batchRange] {
> >>                 // HELP HERE: how to do this ?
> >>                 // essentially I need to map the proper batch of file
> >> names into the call of gj_batch_sim
> >>
> >> simResults[batchSize*i:batchSize*(i+1)-1]=gj_batch_sim(policyFile,
> >> logFile);
> >>         }
> >> }
> >>
> >>
> >>
> >>
> >
> 
> 
> 
>