[Fwd: Re: [Swift-devel] Re: swift-falkon problem... plots to explain plateaus...]

[Mihael Hategan]

Done.

On Tue, 2008-03-25 at 10:16 -0500, Michael Wilde wrote:
> Great, thanks Mihael.  Thats a useful step. I'll test.
> 
> - Mike
> 
> On 3/25/08 10:09 AM, Mihael Hategan wrote:
> > I just wrote a version of the wrapper that opens the log in a descriptor
> > (so opening happens once). I need to test it first, but I'll commit
> > shortly.
> > 
> > On Tue, 2008-03-25 at 10:06 -0500, Michael Wilde wrote:
> >> One thing I'll test is generating the info file on /tmp, and moving it 
> >> when done to the final job dir.
> >>
> >> I can see adjusting wrapper.sh to go from very light to very logged with 
> >> a few increments in the middle that would be most useful.
> >>
> >> The main option I think we want to leave for users to toggle in common 
> >> usage, is whether to run the app with its jobdir on local disk, 
> >> typically below /tmp, or on shared disk.  The user would decide based on 
> >> the job's I/O profile and on local disk space availability.
> >>
> >> Also, I recall some discussion on the success file. Thats acceptable 
> >> overhead for all but the tiniest of jobs, but when a BGP is eventually 
> >> running 100K+ short jobs at once, the rate of success file creation 
> >> could become a bottleneck. Seems like we could have an option that 
> >> avoids creating and expecting the success file if that proved useful - 
> >> need to measure.
> >>
> >> - Mike
> >>
> >>
> >> On 3/25/08 9:32 AM, Mihael Hategan wrote:
> >>> Problem may be that, as a quick test shows, bash opens and closes the
> >>> info file every time a redirect is done.
> >>>
> >>> On Tue, 2008-03-25 at 08:44 -0500, Michael Wilde wrote:
> >>>> I did runs the day before with a modified wrapper that bypassed the INFO 
> >>>> logging. It saved a good amount - I recall about 30% but need to 
> >>>> re-check the numbers.
> >>>>
> >>>> Yes, I came to the same conclusion on the mkdirs.  Im looking at 
> >>>> reducing these, likely moving the jobdir to /tmp.  I think I can do that 
> >>>> within the current structure.  wrapper.sh is ver clear and nicely 
> >>>> written. (Ben: yes, eyeballing the log #s was easy and no problem).
> >>>>
> >>>> First thing I want to do, though, is run some large scale tests on our 
> >>>> two science workflows, increasing the petro-modelling one (the 
> >>>> sub-second application) to a larger runtime through app-level batching.
> >>>>
> >>>> Zhao's latest test indicate that if we do batches of 40, bringing the 
> >>>> jobs from .5 sec to 20 sec, we can saturate the BGP's 4K cores and keep 
> >>>> it running efficiently. Given the extra wrapper.sh overhead, I might 
> >>>> need to increase that another 10X, but once the app is wrapped in a 
> >>>> loop, it makes little difference to the user how big we make that.
> >>>>
> >>>> The other app is a molecule-docking app, that can be batched similarly.
> >>>>
> >>>> Once we get those running nicely at a larger, less brutal job time, I'll 
> >>>> come back to wrapper.sh tuning.  If you or Ben want to do this in the 
> >>>> meantime, though, that would be great.  We have the use-local-disk 
> >>>> scenario on our development stack anyways - this would be a good time to 
> >>>> do it.  If I do it, it will be only a prototype for measurement purposes.
> >>>>
> >>>> Mike
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 3/25/08 8:34 AM, Mihael Hategan wrote:
> >>>>> On Tue, 2008-03-25 at 08:16 -0500, Michael Wilde wrote:
> >>>>>> On 3/25/08 3:31 AM, Mihael Hategan wrote:
> >>>>>>> On Tue, 2008-03-25 at 00:28 -0500, Michael Wilde wrote:
> >>>>>>>> I eyeballed the wrapperlogs to get a rough idea of what was happening.
> >>>>>>>>
> >>>>>>>> I ran with wrapperlog saving and no other changes for wf's of 10, 100 
> >>>>>>>> and 500 jobs, to see how the exec time grew.  At 500 jobs it grew to 
> >>>>>>>> about 30+ seconds for a core app exec time of about 1 sec. (Im just 
> >>>>>>>> recollecting the times as at this point I didnt write much down).
> >>>>>>>>
> >>>>>>> I would personally like to see those logs.
> >>>>>> I listed all the runs in the previous mail (below), Mihael. They are on 
> >>>>>> CI NFS at ~benc/swift-logs/wilde/run{345-350}.
> >>>>> Sorry about that.
> >>>>>
> >>>>>>  Let us know what you find.
> >>>>>>
> >>>>> It looks like this:
> >>>>> - 5 seconds between LOG_START and CREATE_JOBDIR. Likely hogs:
> >>>>> mkdir -p $WFDIR/info/$JOBDIR
> >>>>> mkdir -p $WFDIR/status/$JOBDIR
> >>>>> and the creation of the info file.
> >>>>> - 2.5 seconds between CREATE_JOBDIR and CREATE_INPUTDIR. Likely problem:
> >>>>> mkdir -p $DIR
> >>>>> (on a very fuzzy note, if one mkdir takes 2.5 seconds, two will take 5,
> >>>>> which seems to roughly fit the observed numbers).
> >>>>> - 3.5 seconds for COPYING_OUTPUTS
> >>>>> - 2.5 seconds for RM_JOBDIR
> >>>>>
> >>>>> I'd be curious to know how much of the time is actually spent writing to
> >>>>> the logs. That's because I see one second between EXECUTE_DONE and
> >>>>> COPYING_OUTPUTS, a place where the only meaningful things that are done
> >>>>> are two log messages.
> >>>>>
> >>>>> Perhaps it may be useful to run the whole thing through strace -T.
> >>>>>
> >>>>> Mihael
> >>>>>
> >>>>>
> >>>
> > 
> > 
>