[Swift-devel] Clustering / wrapper.sh logging

[Mihael Hategan]

On Mon, 2007-10-29 at 17:14 +0000, Ben Clifford wrote:
> I just grabbed all the info files that are there at the moment and munged 
> them: (note these are not final results - the workflow continues to run)
> 
> $ cat info.lastsummary 
>  372 END
>    1 EXECUTE
>    1 RM_JOBDIR
> 1017 TOUCH_SUCCESS
> 
> 372 jobs hit the very end of the wrapper script. 1017 of them are a line 
> before that, at TOUCH_SUCCESS state:
> 
> >From wrapper.sh:
> 
> > logstate "TOUCH_SUCCESS"
> > touch status/${ID}-success
> > logstate "END"
> 
> So it looks like I should pay attention first to dealing with status 
> directory scalability.

Here's a bunch of things:
vdl-int.k/execute2
uid := uid()
dprefix := substr(uid, 0, 1)
jobid := concat(tr, "-", uid)
...
task:execute("/bin/sh",
  list("shared/wrapper.sh", jobid, dprefix
    "-e", vdl:executable(tr, rhost), 
...
//update checkStatus
//update transferKickstartRec

wrapper.sh:
...
ID=$1
DPREFIX=$2
checkEmpty "$ID" "Missing job ID"
checkEmpty "$DPREFIX" "Missing directory prefix"
INFO=$WFDIR/info/${DPREFIX}/${ID}-info
...
mkdir -p ../kickstart/${DPREFIX}
...
mv -f kickstart.xml "../kickstart/${DPREFIX}/$ID-kickstart.xml" 2>&1
>>"$INFO"
...
mkdir -p status/${DPREFIX}
touch status/${DPREFIX}/${ID}-success

>