[Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P

[Mihael Hategan]

On Fri, 2009-10-16 at 20:02 -0500, Allan Espinosa wrote:
> The attached graph looks interesting. The green on the left shows the
> 192 job workload using the straight falkon client. The right one is
> through swift.  Using pure falkon the total time for the workload is
> at around 814 seconds while the swift workflow took 1520 seconds.  In
> some repeats of the same run, Swift took 3078 seconds.

It would be useful to plot the swift stuff with swift-plot-log.

It would also be useful to make use of the info files which record
timing data for what the wrapper does.

> 
> Just to confirm, in the deef-provider, the actual jobs that are
> dispatched to the Falkon workers include the ones in the _swiftwrap
> script correct?

_swiftwrap is the job which forks blast (in your case).

>   This must be the one inducing the overheads.  From
> the top of my head create jobdir and link input dir can be the
> culprits.

Sounds plausible.

To that add move-output-files-to-shared-dir. 3 per job as I see it.
Again, info files should provide some details.

> 
> My workflow below does not do any input staging to the work directory.
>  Only output staging is done which I believe is separate from the
> _swiftwrap script and will therefore not register time in the falkon
> graphs (or in EXECUTE state in swift).

Right. Once _swiftwrap is done, the executor should be released, unless
there's some other part in there that takes time (like, say,
notification). Ioan would know this.