[Swift-devel] pilot job paper

[Mihael Hategan]

On Sun, 2011-07-17 at 19:53 -0500, Ketan Maheshwari wrote:
> 
> On Sun, Jul 17, 2011 at 4:23 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         On Sun, 2011-07-17 at 16:19 -0500, Ketan Maheshwari wrote:
>         
>         >
>         >         I'm not sure where cybershake fits in. The problem
>         would be to
>         >         submit
>         >         dummy jobs of various walltimes to various clusters
>         and record
>         >         the
>         >         amount of time they sit in the queue.
>         >
>         > It is just a use-case. I thought, stats based on an (any)
>         real
>         > application will have more merit compared to dummy jobs.
>         >
>         
>         I don't see how it would in this case. We're trying to measure
>         queuing
>         time vs walltime in a general sense. I think it makes more
>         sense to not
>         have a specific application, which would simply be irrelevant
>         information.
> 
> Ok, we can use bionimbus cloud with Mike's catsnsleep script with
> following possible variations:
> 
> Job Completion times: sleep parameter 1s-100s
> Data : cat file size 1M-1G
> Bionimbus VMs 1-32
> 
> Number of jobs n, 1-1000

I need queuing time vs. advertised job walltime on various clusters
(with various/random degrees of utilization). That's to see whether it's
useful to have coasters at all.

The number of jobs is an orthogonal dimension (i.e. we may want to
measure the queuing time vs. #of jobs for various walltimes, but later).
The actual job duration is not relevant. The amount of data is not
relevant.

Clouds are an interesting environment, but not for this particular
problem. That's because we need to see how much it takes to acquire
resources, not how fast some job middleware is after we got hold of
those resources.