[Swift-devel] Re: provider staging stage-in rate on localhost and PADS

[Mihael Hategan]

Ok, so I committed a fix to make the worker send files a bit faster and
adjusted the buffer sizes a bit. There is a trade-off between per worker
performance and number of workers, so this should probably be a setting
of some sort (since when there are many workers, the client bandwidth
becomes the bottleneck).

With a plain cat, 4 workers, 1 job/w, and 32M files I get this:
[IN]: Total transferred: 7.99 GB, current rate: 23.6 MB/s, average rate:
16.47 MB/s
[MEM] Heap total: 155.31 MMB, Heap used: 104.2 MMB
[OUT] Total transferred: 8 GB, current rate: 0 B/s, average rate: 16.49
MB/s
Final status:  time:498988  Finished successfully:256
Time: 500.653, rate: 0 j/s

So the system probably sees 96 MB/s combined reads and writes. I'd be
curious how this looks without caching, but during the run the computer
became laggy, so it's saturating something in the OS and/or hardware.

I'll test on a cluster next.

On Sun, 2011-01-16 at 18:02 -0800, Mihael Hategan wrote:
> On Sun, 2011-01-16 at 19:38 -0600, Allan Espinosa wrote:
> > So for the measurement interface, are you measuring the total data received as
> > the data arrives or when the received file is completely written to the job
> > directory.
> 
> The average is all the bytes that go from client to all the workers over
> the entire time spent to run the jobs.
> 
> > 
> > I was measuring from the logs from JOB_START to JOB_END. I assumed the actualy
> > job execution to be 0. The 7MB/s probably corresponds to Mihael's stage out
> > results.  the cat jobs dump to stdout (redirected to a file in the swift
> > wrapper) probably shows the same behavior as the stageout.
> 
> I'm becoming less surprised about 7MB/s in the local case. You have to
> multiply that by 6 to get the real disk I/O bandwidth:
> 1. client reads from disk
> 2. worker writes to disk
> 3. cat reads from disk
> 4. cat writes to disk
> 5. worker reads from disk
> 6. client writes to disk
> 
> If it all happens on a single disk, then it adds up to about 42 MB/s,
> which is a reasonable fraction of what a normal disk can do. It would be
> useful to do a dd from /dev/zero to see what the actual disk performance
> is.
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel