[Fwd: Re: [Swift-devel] Re: swift-falkon problem... plots to explain plateaus...]

[Mihael Hategan]

On Tue, 2008-03-25 at 08:16 -0500, Michael Wilde wrote:
> On 3/25/08 3:31 AM, Mihael Hategan wrote:
> > On Tue, 2008-03-25 at 00:28 -0500, Michael Wilde wrote:
> >> I eyeballed the wrapperlogs to get a rough idea of what was happening.
> >>
> >> I ran with wrapperlog saving and no other changes for wf's of 10, 100 
> >> and 500 jobs, to see how the exec time grew.  At 500 jobs it grew to 
> >> about 30+ seconds for a core app exec time of about 1 sec. (Im just 
> >> recollecting the times as at this point I didnt write much down).
> >>
> > 
> > I would personally like to see those logs.
> 
> I listed all the runs in the previous mail (below), Mihael. They are on 
> CI NFS at ~benc/swift-logs/wilde/run{345-350}.

Sorry about that.

>  Let us know what you find.
> 

It looks like this:
- 5 seconds between LOG_START and CREATE_JOBDIR. Likely hogs:
mkdir -p $WFDIR/info/$JOBDIR
mkdir -p $WFDIR/status/$JOBDIR
and the creation of the info file.
- 2.5 seconds between CREATE_JOBDIR and CREATE_INPUTDIR. Likely problem:
mkdir -p $DIR
(on a very fuzzy note, if one mkdir takes 2.5 seconds, two will take 5,
which seems to roughly fit the observed numbers).
- 3.5 seconds for COPYING_OUTPUTS
- 2.5 seconds for RM_JOBDIR

I'd be curious to know how much of the time is actually spent writing to
the logs. That's because I see one second between EXECUTE_DONE and
COPYING_OUTPUTS, a place where the only meaningful things that are done
are two log messages.

Perhaps it may be useful to run the whole thing through strace -T.

Mihael