[Swift-devel] Re: provider staging stage-in rate on localhost and PADS

[Mihael Hategan]

Right. So stageouts:
Progress:  time:86039  Selecting site:231  Submitted:1  Active:8
Finished successfully:16
[IN]: Total transferred: 597.1 MB, current rate: 8.99 MB/s, average
rate: 7.02 MB/s
[OUT] Total transferred: 161.33 KB, current rate: 0 B/s, average rate:
1.9 KB/s

That's probably because, as opposed to when the Java side reads files,
there is no read-ahead done by the worker. I'll see if I can add that.

On Sun, 2011-01-16 at 15:21 -0800, Mihael Hategan wrote:
> So I'm running some tests.
> 
> So far here's how the stage-ins look like:
> -site: local (my laptop)
> -job input size: 32MB
> -256 jobs
> -job output size 0B
> -measurement is made at the interface between service and worker (the
> TCP connection). It is aggregated for all workers.
> -2 workers, 4 jobs per worker
> -this is the fast branch, but the job throughput is pretty irrelevant
> here since this is I/O bound.
> 
> What I get is this:
> file:
> [IN]: Total transferred: 641.93 KB, current rate: 0 B/s, average rate:
> 4.62 KB/s
> [OUT] Total transferred: 8 GB, current rate: 40.2 MB/s, average rate:
> 58.92 MB/s
> Final status:  time:140732  Finished successfully:256
> Time: 142.13, rate: 1 j/s
> 
> proxy:
> Final status:  time:113915  Finished successfully:256
> Time: 115.393, rate: 2 j/s
> [IN]: Total transferred: 705.62 KB, current rate: 6.44 KB/s, average
> rate: 6.24 KB/s
> [OUT] Total transferred: 8 GB, current rate: 36.08 MB/s, average rate:
> 72.54 MB/s
> 
> (It is funny that proxy is faster than "file", but for now I'll ignore
> that).
> 
> For comparison:
> mike at blabla2 coasters$ time dd if=/dev/zero of=~/tmp/8g bs=32KB
> count=262144
> 262144+0 records in
> 262144+0 records out
> 8388608000 bytes (8.4 GB) copied, 49.9836 s, 168 MB/s
> 
> Things could be improved, but staging in does not seem to be the
> bottleneck. 
> Next, stage-outs...
> 
> Mihael
> 
> 
> 
> 
> On Wed, 2011-01-12 at 17:05 -0800, Mihael Hategan wrote:
> > On Wed, 2011-01-12 at 15:57 -0600, Daniel S. Katz wrote:
> > > As I read this, it appears that there is some limit in Swift, rather than in the hardware, that is causing these numbers to be very low.
> > > 
> > > Mihael, do you agree?
> > 
> > Yes, but that does not exclude a limit in hardware in other
> > configurations.
> > 
> > >   Can you help us figure out what's going on?
> > 
> > Of course.
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel