[Swift-devel] Re: Drop kickstart, adopt wrapper logs?

[Michael Wilde]

Im happy with wrapper for now.

Regarding use: there is a real need. *I* used kickstart quite a bit when 
working on apps. It doesnt get used because its not easy and not efficient.

A real user need exists right now: the OOPS folding app has memory and 
cpu usage issues that impact its feasibility. The user (Glen Hocky) 
needs to compare its run time on different platforms with different 
compilers, and also needs to tune how many copies run on multicore hosts 
(bgp, sicortex, Abe/QueenBee on TG, and Ranger - 4, 6, 8, and 16 
respectively).

So Glen is essentially doing his own logging in the app wrapper, using 
/usr/bin/time.  But this is something we can and should do for the user, 
simply, naturally, and by default. And make it easy enough that the 
basic app performance characteristics are reported on every run.

Very similar needs exist with DOCK: we have some mysterious app failures 
that we'd like to correlate with mem consumption.

And BLAST: Ive told Alina to to ad-hoc time/mem recording there as well.

In the BGP we are starting to discuss collective I/O, and have been 
running apps in a way that returns all their little output files - 
including ad-hoc time reports, in a single tarball per job.

So I think this is worth discussing and improving.

I'll take a look at what wrapper is doing, and we can discuss a few 
options for moving the logs. One of which is that the user bundles it 
explicitly. Another is that the files are streamed and/or batched. A 
third is that the info is stream as Mihael suggests.

- Mike


On 7/25/08 10:28 AM, Mihael Hategan wrote:
> In the really long run if coasters turn out to be a workable piece of
> software, they could support remote logging, which would mean streaming
> logging data instead of using the filesystem.
> 
> But for the short term, I agree. We haven't exercised kickstart as much,
> and the wrapper logs seem to be reasonably useful as a replacement.
> 
> On Fri, 2008-07-25 at 15:15 +0000, Ben Clifford wrote:
>> pretty much no one uses kickstart with swift at the moment. The wrapper 
>> log *is* in a useful format, and is probably as specified as any other 
>> part of the runtime environment.
>>
>>
>>> was: Re: Can we copy Swift execution logs to CI network?
>>>
>>> Should we make this transition, dropping kickstart, and making the wrapper log
>>> a useful format that becomes part of the Swift runtime specification?
>>>
>>> On 7/15/08 4:18 PM, Ben Clifford wrote:
>>> <snip>
>>>> ...there's substantial lack of information
>>>> for many runs as we have been tweaking the logs over time (especially the
>>>> worker node logs which take a similar place to kickstart records now -
>>>> giving the actual on-worker cpu usage but are relatively very expensive to
>>>> collect and move around).
>>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>