[Swift-devel] Re: provider staging stage-in rate on localhost and PADS

[Mihael Hategan]

On Sun, 2011-01-16 at 19:38 -0600, Allan Espinosa wrote:
> So for the measurement interface, are you measuring the total data received as
> the data arrives or when the received file is completely written to the job
> directory.

The average is all the bytes that go from client to all the workers over
the entire time spent to run the jobs.

> 
> I was measuring from the logs from JOB_START to JOB_END. I assumed the actualy
> job execution to be 0. The 7MB/s probably corresponds to Mihael's stage out
> results.  the cat jobs dump to stdout (redirected to a file in the swift
> wrapper) probably shows the same behavior as the stageout.

I'm becoming less surprised about 7MB/s in the local case. You have to
multiply that by 6 to get the real disk I/O bandwidth:
1. client reads from disk
2. worker writes to disk
3. cat reads from disk
4. cat writes to disk
5. worker reads from disk
6. client writes to disk

If it all happens on a single disk, then it adds up to about 42 MB/s,
which is a reasonable fraction of what a normal disk can do. It would be
useful to do a dd from /dev/zero to see what the actual disk performance
is.