[Swift-devel] Drop kickstart, adopt wrapper logs?

[Michael Wilde]

If you do take this off for weekend work (which sounds like a good idea, 
modulo the weekend part ;) lets first set a design decision making 
process on the issues that have been raised in discussions to date:

- arg passing
- what it captures
   - time, mem
   - env (in the broad sense)
   - file state
- output format(s)
- snapshotting for monitoring an app run
- controls for start, stop, pause
- how it integrates in CoG, GRAM, Condor, other LRMs, Coaster, Falkon, 
and Swift
- how it relates to "accounting" projects in OSG and TG
- ideas on streaming output
- idead on batching output
- interaction with a collective I/O toolkit/model
- other stuff???

Its worth looking at what Jens did in "kickstart v2", what he called 
"k2", in the VDS CVS.  It had conventions to deal with passing args in a 
file stream rather than command line to tunnel through what at the time 
were cli length limits and arg escape problems in Condor

I feel this merits a design effort, and propose that we try this as an 
experiment in doing design in a way that can get group input do decision 
making in an effective and pleasing way.

Im thinking email discussion but based on a persistent changing document 
(bugzilla note, wiki page, or even text in email that we treat as a 
doc). Something that when we are done discussing and say "go", we have a 
single place that documents the complete consensus to that point.

- Mike

On 7/25/08 10:34 AM, Mihael Hategan wrote:
> Could be a decent weekend hack for me though.
> 
> On Fri, 2008-07-25 at 15:20 +0000, Ben Clifford wrote:
>> Its on my todo list. Its below the event horizon though (that being the 
>> line at which more prioritised things are arriving at a rate such that 
>> this never rises and is instead sucked into the black hole of infinity) so 
>> it won't get done. 
>>
>> On Fri, 25 Jul 2008, Ian Foster wrote:
>>
>>> Mike:
>>>
>>> There was an agreement a year or more ago that the VDS group would work with
>>> the GRAM team on integration. I suspect it got dropped, but I wanted to check.
>>>
>>> Ian.
>>>
>>> On Jul 25, 2008, at 8:17 AM, Michael Wilde wrote:
>>>
>>>> Im not aware of any work done in that direction.
>>>> I would have to hunt back through emails for the last thing we said we would
>>>> do.
>>>>
>>>> Its been an often discussed topic, but one thats hard to move forward on.
>>>>
>>>> The tradeoff is doing something that works well for swift, vs having a
>>>> significantly larger and longer discussion on what would work for GRAM as
>>>> well (and potentially Condor and ...)
>>>>
>>>> One possibility is that we stay with a kickstart like architecture where
>>>> kickstart is the last thing called to launch the app, and that becomes a
>>>> separable component that can be used elsewhere. We do it in a way that its
>>>> useful with GRAM and other systems, and which users can just use.
>>>>
>>>> Then the discussion becomes mostly one of data format.
>>>> There the two likely candidates are XML or name/value pairs, possibly in
>>>> some other "standard" format, eg classads.
>>>>
>>>> Another possibility is to emit an xml doc in a rigid format that doesnt need
>>>> an xml parser to process it - sort of n/v pairs in xml. This gets
>>>> complicated with multiline and escape issues of course.
>>>>
>>>> Its worth a chat with the GRAM team.  It will be a slower harder route
>>>> though.  Thats a tradeoff.
>>>>
>>>> - Mike
>>>>
>>>>
>>>> On 7/25/08 8:01 AM, Ian Foster wrote:
>>>>> Mike:
>>>>> What is the status of the integration into kickstart into the Globus
>>>>> release?
>>>>> Ian.
>>>>> On Jul 25, 2008, at 7:41 AM, Michael Wilde wrote:
>>>>>> was: Re: Can we copy Swift execution logs to CI network?
>>>>>>
>>>>>> Should we make this transition, dropping kickstart, and making the
>>>>>> wrapper log a useful format that becomes part of the Swift runtime
>>>>>> specification?
>>>>>>
>>>>>> On 7/15/08 4:18 PM, Ben Clifford wrote:
>>>>>> <snip>
>>>>>>> ...there's substantial lack of information
>>>>>>> for many runs as we have been tweaking the logs over time (especially
>>>>>>> the worker node logs which take a similar place to kickstart records
>>>>>>> now - giving the actual on-worker cpu usage but are relatively very
>>>>>>> expensive to collect and move around).
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel