[Swift-user] Workflow handling a large parallelism ?

[Tiberiu Stef-Praun]

Hi

I need to build a fairly simple workflow (fan-in structure) that has a
relatively large number of parallel tasks (~1000).
Can you help me with solution smarter than the one below ?

//note, code is not debugged, will probably not run as is, however,
this is how I would code it up in swift

type file{}
(file simOut)run_sim(int n){
    app{ run_sim n stdout=@filename(simOut) ; }
}

(file simMerged)merge_sim(file simFiles[]){
    app{ cat @filenames(simFiles) ; }
}

(file simFiles[]) batch_sim (){

    int simRange = [1:1000];
    forach i in simRange {
        simFiles[i]=run_sim(i);
    }
}

//I am concerned about this, I would like to be able to generate the
filenames in swift,
//not to be forced to list all the names

string filenames[] = [ "sim_000",'sim_001", ...  "sim_999" ];
file simOutputs[] <array_mapper; files=filenames>;

filenames = batch_sim();

file mergedSim<single_file_mapper; file="merged.out">;
mergedSim = merge_sim(simOutputs);


-- 
Tiberiu (Tibi) Stef-Praun, PhD
Research Staff, Computation Institute
5640 S. Ellis Ave, #405
University of Chicago
http://www-unix.mcs.anl.gov/~tiberius/