[Swift-devel] Analysis of wrapper.sh

[Zhao Zhang]

Hi, All

I made this analysis of wrapper.sh. Correct me, if there is anything 
wrong. Thanks.

zhao


SWIFT phase: When swift is started, it creates a directory with the 
workload name and a random string, something like

/home/zzhang/swift/sleep-20080724-1527-qakbkkcc
                                                        |
                                                        |____info
                                                        |
                                                        |____kickstart
                                                        |
                                                        |____shared
                                                        |            |
                                                        |            
|____wrapper.sh
                                                        |            |
                                                        |            
|____seq.sh
                                                        |
                                                        |____status


WRAPPER.SH phase

In my test case, the BGexec received a task in such a format
"shared/wrapper.sh sleep-l5clzzvi -jobdir l -e /bin/sleep -out 
stdout.txt -err stderr.txt -i -d  -if  -of  -k  -a 600"
with the working dir "/home/zzhang/swift/sleep-20080724-1527-qakbkkcc"

WORKING_DIRECTORY                                                 OPERATION
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            OPEN wrapper.log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK if 
jobid ($1, sleep-l5clzzvi ) is empty    ----> empty exit with 254
                                                                        
                                        |
                                                                      
                                          |
                                                                      
                                         V
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            Get -jobdir 
as $JOBDIR
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK if 
-jobdir ( l )is empty      ----> empty exit with 254
                                                                       
                                         |
                                                                      
                                          |
                                                                      
                                         V
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            mkdir -p 
$WFDIR/info/$JOBDIR   (mkdir 
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc/info/l )
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            rm -f 
"$WFDIR/info/$JOBDIR/${ID}-info" ( make a clean $ID-info file 
ID=sleep-l5clzzvi )
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            openinfo 
"$WFDIR/info/$JOBDIR/${ID}-info"
                                                                      
                       ( openinfo 
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc/info/l/sleep-l5clzzvi-info)
                                                                        
                      creating log file

/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"LOG_START" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT "Wrapper" 
into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc               mkdir -p 
$WFDIR/status/$JOBDIR (create status parent dir for the job "mkdir -p 
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc/status/l ")
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PARSE the 
arguments
                                                                        
                     ( EXEC=/bin/sleep, STDOUT=stdout.txt, 
STDERR=stderr.txt, STDIN=NULL,
                                                                        
                        DIRS=null, INF=NULL, OUTF=NULL, KICKSTART=NULL)
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            Check if 
there are arguments after -a  ----> empty exit with 254
                                                                       
                                         |
                                                                      
                                          |
                                                                      
                                         V
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            change $@ 
from "-a 600" to "600"
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            Check if 
"$SIWFT_JOBDIR_PATH" is NULL ----> NO, local copy
                                                                       
                                         |
                                                                      
                                          |
                                                                      
                                         V
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            YES, shared 
file system. DIR=jobs/$JOBDIR/$ID ( DIR=jobs/l/sleep-l5clzzvi )
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            set PATH
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT all 
arguments into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"CREATE_JOBDIR" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            mkdir -p $DIR 
(In the working dir, "mkdir -p jobs/l/sleep-l5clzzvi ")
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK if 
"mkdir" is successful  ----> NO, exit with 254
                                                                       
                                         |
                                                                      
                                          |
                                                                      
                                         V
                                                                        
                     YES, put "Created job directory : $DIR" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"CREATE_INPUTDIR" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            Created all 
subdirs in $DIR as in $DIRS ( create the same tree in 
jobs/l/sleep-l5clzzvi as in input file dir)
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"LINK_INPUTS" in to log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK file 
system type  ----> local disck, cp all files in $PWD/shared to $DIR
                                                                      
                                          |
                                                                      
                                          |
                                                                      
                                         V
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            shared file 
system, create links in  $DIR for all files in $PWD/shared
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT "EXECUTE" 
into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK if 
"kickstart is enabled" ----> yes, use kickstart to run the job
                                                                      
                                          |
                                                                      
                                          |
                                                                      
                                         V
                                                                        
                     NO, run the job with wrapper.sh
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"EXECUTE_DONE" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT "Job ran 
successfully" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            CHECK if the 
out put dir tree is the same as the one in $OUTF ----> NO, exit with 254
                                                                      
                                          |
                                                                      
                                          |
                                                                      
                                         V
                                                                        
                    YES, COPY all output files in $DIR back to $PWD/shared
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"RM_JOBDIR" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            rm -rf $DIR 
("rm -rf jobs/l/sleep-l5clzzvi ")
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT 
"TOUCH_SUCCESS" into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            touch 
status/${JOBDIR}/${ID}-success ( touch status/l/)sleep-l5clzzvi-success )
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            PUT "END" 
into log
/home/zzhang/swift/sleep-20080724-1527-qakbkkcc            closeinfo