[Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P

[Allan Espinosa]

The attached graph looks interesting. The green on the left shows the
192 job workload using the straight falkon client. The right one is
through swift.  Using pure falkon the total time for the workload is
at around 814 seconds while the swift workflow took 1520 seconds.  In
some repeats of the same run, Swift took 3078 seconds.

Just to confirm, in the deef-provider, the actual jobs that are
dispatched to the Falkon workers include the ones in the _swiftwrap
script correct?  This must be the one inducing the overheads.  From
the top of my head create jobdir and link input dir can be the
culprits.

My workflow below does not do any input staging to the work directory.
 Only output staging is done which I believe is separate from the
_swiftwrap script and will therefore not register time in the falkon
graphs (or in EXECUTE state in swift).
workflow:
ype output;
type std_err;
type std_out;

type BlastDatabase;
type BlastQuery;

type BlastResult {
  output out;
  std_err serr;
  std_out sout;
}

type BlastSummary;

app (BlastResult out) blastall(string i, string db) {
  blastall "-p" "blastp" "-m" "8" "-e" "1.0e-5" "-FF"
      "-d" db "-i" i stdout=@filename(out.sout)
      "-o" @filename(out.out) stderr=@filename(out.serr);
}

string db="/intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/nr_v200/nr";

BlastResult out[] <ext;exec="mapper-out.sh">;
BlastQuery input[] <ext;exec="mapper-in.sh">;

foreach data,i in input {
  out[i] = blastall(@strcat("/", at filename(data)), db);
}


The build of swift i'm using is fairly old (Swift svn swift-r2911 cog-r2394)

-Allan
-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: summary_graph.png
Type: image/png
Size: 5305 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20091016/49d16144/attachment.png>