[Swift-devel] Re: provider staging stage-in rate on localhost and PADS

[Michael Wilde]

Mihael,

If I follow this right, your stageout rate (7MB/sec) is about what Allan saw for stage ins (~ 6MB/sec as I recall).

But your stagein rate is 59MB/sec and 73MB/sec depending on the mode (file vs proxy).

So that does not yet replicate what he's seeing (on the WAN).

Allan, I cant recall if you sent numbers for local-host tests?

- Mike




----- Original Message -----
> Right. So stageouts:
> Progress: time:86039 Selecting site:231 Submitted:1 Active:8
> Finished successfully:16
> [IN]: Total transferred: 597.1 MB, current rate: 8.99 MB/s, average
> rate: 7.02 MB/s
> [OUT] Total transferred: 161.33 KB, current rate: 0 B/s, average rate:
> 1.9 KB/s
> 
> That's probably because, as opposed to when the Java side reads files,
> there is no read-ahead done by the worker. I'll see if I can add that.
> 
> On Sun, 2011-01-16 at 15:21 -0800, Mihael Hategan wrote:
> > So I'm running some tests.
> >
> > So far here's how the stage-ins look like:
> > -site: local (my laptop)
> > -job input size: 32MB
> > -256 jobs
> > -job output size 0B
> > -measurement is made at the interface between service and worker
> > (the
> > TCP connection). It is aggregated for all workers.
> > -2 workers, 4 jobs per worker
> > -this is the fast branch, but the job throughput is pretty
> > irrelevant
> > here since this is I/O bound.
> >
> > What I get is this:
> > file:
> > [IN]: Total transferred: 641.93 KB, current rate: 0 B/s, average
> > rate:
> > 4.62 KB/s
> > [OUT] Total transferred: 8 GB, current rate: 40.2 MB/s, average
> > rate:
> > 58.92 MB/s
> > Final status: time:140732 Finished successfully:256
> > Time: 142.13, rate: 1 j/s
> >
> > proxy:
> > Final status: time:113915 Finished successfully:256
> > Time: 115.393, rate: 2 j/s
> > [IN]: Total transferred: 705.62 KB, current rate: 6.44 KB/s, average
> > rate: 6.24 KB/s
> > [OUT] Total transferred: 8 GB, current rate: 36.08 MB/s, average
> > rate:
> > 72.54 MB/s
> >
> > (It is funny that proxy is faster than "file", but for now I'll
> > ignore
> > that).
> >
> > For comparison:
> > mike at blabla2 coasters$ time dd if=/dev/zero of=~/tmp/8g bs=32KB
> > count=262144
> > 262144+0 records in
> > 262144+0 records out
> > 8388608000 bytes (8.4 GB) copied, 49.9836 s, 168 MB/s
> >
> > Things could be improved, but staging in does not seem to be the
> > bottleneck.
> > Next, stage-outs...
> >
> > Mihael
> >
> >
> >
> >
> > On Wed, 2011-01-12 at 17:05 -0800, Mihael Hategan wrote:
> > > On Wed, 2011-01-12 at 15:57 -0600, Daniel S. Katz wrote:
> > > > As I read this, it appears that there is some limit in Swift,
> > > > rather than in the hardware, that is causing these numbers to be
> > > > very low.
> > > >
> > > > Mihael, do you agree?
> > >
> > > Yes, but that does not exclude a limit in hardware in other
> > > configurations.
> > >
> > > >   Can you help us figure out what's going on?
> > >
> > > Of course.
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory