[Swift-devel] Formatting kickstart records

[Mihael Hategan]

That document isn't very clear about what falls under "MUST" and what is
"recommended".

There is no need for keywords when a field is mandatory and the position
is well defined. For how some programming languages deal with the issue
of mixing mandatory and optional arguments (using position and keyword),
see:
http://docs.python.org/ref/calls.html#calls
http://reference.wolfram.com/mathematica/tutorial/SettingUpFunctionsWithOptionalArguments.html
http://www.tc.cornell.edu/Services/Education/Topics/General/Fortran90/7.1+Keyword+and+Optional+Arguments.htm
http://caml.inria.fr/pub/docs/u3-ocaml/ocaml051.html
http://www.adahome.com/rm95/rm9x-06-04.html
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node64.html#SECTION00922000000000000000

I can hardly tell where the date ends and the time begins. As such, at
least from my perspective, the claim that the recommended timestamp
format is "highly readable" is suspicious. Perhaps the document could
abstain from making religious statements.

Mihael

On Tue, 2007-09-11 at 09:47 -0700, Brian Tierney wrote:
> Hi Mike:
> 
> The logging document is here:
> 
> http://www.cedps.net/wiki/index.php/LoggingBestPractices
> 
> and a paper describing the higher-level vision is here:
> 
> http://www.cedps.net/wiki/images/e/ec/Grid2007.pdf
> 
> It should be quite easy to convert your logs to the CEDPS format.
> 
> Let's read each others papers and then talk more.
> 
> 
> 
> 
> Michael Wilde wrote:
> > Brian, Suchandra,
> > 
> > In the Swift workflow system we use a tool (developed as part of the 
> > GriPhyN Virtual Data System VDS) called "kickstart" to launch programs 
> > on Grid sites and record their execution environment, exit status and 
> > resource consumption as an XML document. Kickstart is a simple utility 
> > runs the real app as a child process and records its info; it usually 
> > run under GRAM.
> > 
> > We're now developing some tools to flexibly format and and query this 
> > info, as its very useful in troubleshooting.
> > 
> > Ian suggests below that we should integrate this mechanism with a 
> > CEDPS-defined log format. Ive looked at the CEDPS wiki, but have not yet 
> > found where this format is defined and how it relates to the myriad 
> > sources of gird log info.
> > 
> > Could you point us to some information on this format and how you 
> > envision that tools like kickstart would integrate with it?
> > 
> > Any thoughts you have on how we should approach logging in Swift would 
> > be welcome as well.
> > 
> > For reference: Swift is described at www.ci.uchicago.edu/swift;
> > A sample kickstart record is at:
> > http://www.ci.uchicago.edu/~wilde/runs/uc.teragrid/runs/run23/kickstart/angle4-0aac5zgi-kickstart.xml 
> > 
> > A paper on kickstart is at:
> > http://www.ci.uchicago.edu/swift/papers/Kickstarting2006.pdf
> > 
> > Thanks,
> > 
> > Mike
> > 
> > 
> > Ian Foster wrote:
> >> is this format in the "standard log format" that CEDPS defined? I 
> >> would think it should be.
> >>
> >> Michael Wilde wrote:
> >>> I started a small perl script to format kickstart records.
> >>>
> >>> For now, for each kickstart output file on the cmd line it prints 
> >>> one-liners of the form:
> >>>
> >>> Start=2007-09-09T00:24:43.161-05:00 duration=83.993 user=82.470 
> >>> sys=1.360 machine=i686 host=tg-v050.uc.teragrid.org
> >>>
> >>> but its pretty generalizable.
> >>>
> >>> I did this to find the min, max and stats on run times, looking for 
> >>> outliers that are holding up the workflow.
> >>>
> >>> If anyone has a similar/better tool please point it out, otherwise 
> >>> I'll continue to enhance this. Suggestions for a better approach are 
> >>> welcome.
> >>>
> >>> Pavel (summer student) did something like this in C a while back; I 
> >>> thought that was added to vds/contrib at that time but I dont see it 
> >>> in the latest VDS release.  Need to hunt it down.
> >>>
> >>> This perl script is very simple and easy, but rather slow on 1000 
> >>> kickstart records (need to get timings; Im sure it can be improved, 
> >>> possibly by grabbing multiple fields on each XPath call.
> >>>
> >>> - Mike
> >>>
> >>> $ cat ~/vds/kix
> >>> #!/usr/bin/perl -w -I/home/wilde/vds
> >>> #                  ^^^ How to best set the module path?
> >>>
> >>> # Print fields from a list of invocation record xml files
> >>>
> >>> use strict;
> >>> use XML::XPath;
> >>> use XML::XPath::XMLParser;
> >>>
> >>> while(@ARGV) {
> >>>   print_irec(shift @ARGV);
> >>> }
> >>>
> >>> sub print_irec
> >>> {
> >>>   my $irec = shift;
> >>>   my $xp = XML::XPath->new(filename => $irec);
> >>>
> >>>   my $start = $xp->findvalue('/invocation/mainjob/@start');
> >>>   my $utime = $xp->findvalue('/invocation/mainjob/usage/@utime');
> >>>   my $stime = $xp->findvalue('/invocation/mainjob/usage/@stime');
> >>>   my $duration = $xp->findvalue('/invocation/mainjob/@duration');
> >>>   my $machine = $xp->findvalue('/invocation/uname/@machine');
> >>>   my $host = $xp->findvalue('/invocation/@hostname');
> >>>
> >>>   print "Start=$start duration=$duration user=$utime sys=$stime 
> >>> machine=$machine host=$host\n";
> >>> }
> >>>
> >>> _______________________________________________
> >>> Swift-devel mailing list
> >>> Swift-devel at ci.uchicago.edu
> >>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> >>>
> >>
>