[Swift-devel] jobchart

[Yong Zhao]

In Swift there is dir creation, exit code checking, cleanup etc. but that
is still a lot of extra time (25s vs 3s), kinda weird

Yong.

On Sat, 10 Mar 2007, Mihael Hategan wrote:

> On Sat, 2007-03-10 at 23:16 -0600, Mihael Hategan wrote:
> > On Sat, 2007-03-10 at 23:14 -0600, Mihael Hategan wrote:
> > > On Sat, 2007-03-10 at 23:02 -0600, Mihael Hategan wrote:
> > > > On Sat, 2007-03-10 at 23:02 -0600, Yong Zhao wrote:
> > > > > I see, so this does not involve stage-in and outs. Why did postprocessing
> > > > > take that much time in the other two graphs, where in this case it is
> > > > > minimal?
> > > >
> > > > I think that's a cpu that does too many things at once. But I welcome
> > > > other explanations.
> > >
> > > Actually I'm kinda confused. Running bash with 300 parallel "/bin/sleep
> > > 2" takes 2.4 seconds and doing it with plain Karajan takes 5.
> >
> > Also, running 300 parallel wait=2000 in karajan takes 2.08 seconds. So
> > the bulk must be in the implementation of execute() and/or the local
> > provider.
>
> Note: not counting the jvm startup:
> print(time(parallelfor(i, range(1, 300), wait(delay=2000)))
>
> >
> > >
> > > Some thinking needs to happen there.
> > >
> > > >
> > > > >
> > > > > Yong.
> > > > >
> > > > > On Sat, 10 Mar 2007, Mihael Hategan wrote:
> > > > >
> > > > > > Preprocessing is the time between "Running job..." log message in
> > > > > > vdl-int.k and when the task status gets changed to "Submitted".
> > > > > >
> > > > > > Postprocessing is between when the task status is changed to "Completed"
> > > > > > and the "Completed job" log message in vdl-int.k, and includes the check
> > > > > > for the exit code file.
> > > > > >
> > > > > > Mihael
> > > > > >
> > > > > > On Sat, 2007-03-10 at 22:51 -0600, Yong Zhao wrote:
> > > > > > > nice graph, what's preprocessing and postprocessing involved in this case?
> > > > > > >
> > > > > > > Yong.
> > > > > > >
> > > > > > > On Sat, 10 Mar 2007, Mihael Hategan wrote:
> > > > > > >
> > > > > > > > There's a new tool in bin.
> > > > > > > >
> > > > > > > > It's a spin off Jens' "show-id" tool.
> > > > > > > > After careful analysis of show-id, it became apparent that a lot of the
> > > > > > > > difficulty was in gathering and organizing the data, rather than in
> > > > > > > > generating the plots. This one's written in python and lacks the command
> > > > > > > > line options to control sizes, but includes the logic in Jens' tool that
> > > > > > > > automatically scale things.
> > > > > > > >
> > > > > > > > It does not show individual stage-ins and stage-outs. I'll have to think
> > > > > > > > of a way to represent those on the plot without making it messy.
> > > > > > > > It needs the logs to contain debugging info from individual tasks:
> > > > > > > > log4j.logger.org.globus.cog.abstraction.impl.common.task.TaskImpl=DEBUG
> > > > > > > >
> > > > > > > > I've updated this in SVN, but if you want to run it on older builds, you
> > > > > > > > need the above in log4j.properties.
> > > > > > > >
> > > > > > > > I attached a sample output.
> > > > > > > >
> > > > > > > > Mihael
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > > >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >
>
>