[Swift-devel] How to increase Swift-coaster task rate

[Michael Wilde]

We need to re-run the tests on a machine we can get to. The current tests were run for us by a Cray engineer, and we havent tarred the logs back yet.

Submit host machine was a 6-core Cray login host, probably AMD at 2.5GHz or so, probably 32GB RAM.

Coaster service was local in Swift jvm. Can try a separate service.

Java was (I think) Sun 1.6.0_21, IBM Java is there. Not sure if Java was 64 bit.

Jon will re-run clean tests on the Raven Cray where we have full access.

Advice on dialing the logging way down would be valuable. Outside of that we're trying to avoid all ancillary touches of shared filesystems.

Thanks,

- Mike


----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, April 26, 2012 8:47:38 PM
> Subject: Re: How to increase Swift-coaster task rate
> Can I see the log?
> 
> I don't think that there is a set of things that I can easily point
> out,
> but I can try to see what the problems might be.
> 
> What's the submit host (cpu/cores/mem, etc)?
> 
> Separate service or auto/local?
> 
> Mihael
> 
> On Thu, 2012-04-26 at 20:37 -0500, Michael Wilde wrote:
> > Mihael,
> >
> > David, Jon, and I are working on Cray benchmarks for a paper for the
> > Cray Users Group.
> >
> > In tests so far, we are being limited by job submission rates of
> > about 80 tasks/sec.
> >
> > We'd like very much to drive that up closer to 200/sec if at all
> > possible for the benchmarks we're trying to run.
> >
> > The current tests are doing sleep 0 jobs with no file transfer to
> > about 2400 cores on a Cray benchmark system. The workdir is set to
> > /dev/shm. The throttles are almost all set way up (Jon can post the
> > specific config and values).
> >
> > One thing we have not yet done is try to get the log traffic way
> > down; thats next up to try.
> >
> > We'll revert to testing against 480 cores on raven for now. That
> > should still be enough to push the upper limit of Swift, Karajan and
> > coasters.
> >
> > Can you give us a set of things to check (set, turn off, etc) to try
> > to get closer to 200 tasks/sec? Do we need to set <scratch> to
> > /dev/shm in addition to work dir?
> >
> > This latest run was I think with provider staging and pin coaster
> > files.
> >
> > Thanks,
> >
> > - Mike
> >
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory