[Swift-user] Help needed with batching up parallel runs

[Tiberiu Stef-Praun]

Hi, thanks, I forgot about that.
I tried running it on teraport, and it failed.
The log is here:
http://www.ci.uchicago.edu/~tiberius/issues/gj-batched-20080723-1522-fypk29g6.log

Mihael suggested that I should file a bug report, but I'm not sure
what to report (other than the failure)

Tibi

On Tue, Jul 22, 2008 at 2:14 AM, Ben Clifford <benc at hawaga.org.uk> wrote:
>
> Use clustering. Read the docs mike linked to. Basically you need to
> specify a maxwalltime for the jobs you want clustered, and then a
> clustering time that is some multiple (eg 10 in your case).
>
> You might try coasters if you are submitting using GT2 (there is something
> wrong with gt4 + coasters at the moment that prevents them being used
> together).
>
> On Mon, 21 Jul 2008, Tiberiu Stef-Praun wrote:
>
>> Hi
>>
>> I work with some code that generates at some point a number (300 in my
>> case) of parallel identical runs, and I need to batch those up (10 at
>> a time in my case) because each individual run is too short.
>> I don't want Falkon at this point, and I'm not sure about the status
>> of the coaster provider, so I would prefer a clean swift solution
>> I was thinking of some array manipulation, but it was not obvious how
>> to do it with swift.
>>
>> Thanks !
>> Tibi
>>
>> Here is the code that I have so far, and I need help for:
>>
>>
>>
>> //this is the code that batches a number of runs: based on the size of
>> the array (determined where I make the call), I will return the set of
>> parallel run results
>> (file simFile[])gj_batch_sim(file policyFile,  file logFile){
>>         app{
>>                 gj_batch_sim @filename(policyFile) @filename(logFile)
>> @filenames(simFile);
>>         }
>> }
>>
>> int parallelInstances=300;
>> file simOutputs[]<simple_mapper; prefix="sim_", suffix=".mat">;
>>
>> (file simResults[])batch_gj_batch_sim(file policyFile, int parallelInstances){
>>         // this is just some needed input
>>         file logFile<single_file_mapper; file="inputs/gj_log76.mat">;
>>
>>        // I want to have batches of size 10
>>         int localBatchSize=10;
>>
>>         int batchRange=@toint(@strcut(@strcat(parallelInstances/localBatchSize),"([0-9]*).?[0-9]*")
>>         trace("Times to do batch_gj_batch_sim",batchRange);
>>
>>         foreach i in  [1:batchRange] {
>>                 // HELP HERE: how to do this ?
>>                 // essentially I need to map the proper batch of file
>> names into the call of gj_batch_sim
>>
>> simResults[batchSize*i:batchSize*(i+1)-1]=gj_batch_sim(policyFile,
>> logFile);
>>         }
>> }
>>
>>
>>
>>
>



-- 
Tiberiu (Tibi) Stef-Praun, PhD
Computational Sciences Researcher
Computation Institute
5640 S. Ellis Ave, #405
University of Chicago
http://www-unix.mcs.anl.gov/~tiberius/