[Swift-devel] How does swift know if a task is successful

[skenny at uchicago.edu]

hey zhao, did you get this to work? was thinking i might try
it on ranger, but i was wondering if you also then have to
hack something else to prevent swift from cleaning up your
work directory? that is, i assume you actually DO want the
output, you just don't want to have to wait for the stageouts (?)

---- Original message ----
>Date: Tue, 17 Mar 2009 13:40:30 -0500
>From: Mihael Hategan <hategan at mcs.anl.gov>  
>Subject: Re: [Swift-devel] How does swift know if a task is
successful  
>To: Zhao Zhang <zhaozhang at uchicago.edu>
>Cc: swift-devel <swift-devel at ci.uchicago.edu>
>
>On Tue, 2009-03-17 at 13:36 -0500, Zhao Zhang wrote:
>> Hi, Mihael
>> 
>> I commented the following lines
>>   /*dir:make(ldir)
>>      restartOnError(".*", 2
>>      task:transfer(srchost=host, srcfile=bname,
>>      srcdir=rdir, destdir=ldir, desthost=dhost,
destprovider=provider)
>>   )*/
>> 
>
>Did you modify this file in dist/?/libexec? If not, did you
re-compile
>swift after the modification?
>
>Put an echo or a log message in place, to see if your change
is picked
>up by swift next time.
>
>> Then I modified wrapper.sh to not to copy output file back,
but I still 
>> got an error.
>> The log file is at 
>>
http://www.ci.uchicago.edu/~zzhang/first-20090317-1327-oqgttus8.log
>> Thanks
>> 
>> zhao
>> 
>> zzhang at login6.surveyor:~/new_dock6> ./run_swift_ssh.sh 1117
64 first.swift
>> waiting for at least 64 nodes to register before submitting
workload...
>> waiting to find at least 1 services in file 
>>
/home/falkon/users/zzhang/1117/config/Client-service-URIs.config...
>> all done, file has found at least 1 services
>> found at least 64 registered, submitting workload...
>> Swift svn swift-r2676 (swift modified locally) cog-r2305
>> 
>> RunID: 20090317-1327-oqgttus8
>> Progress:
>> Progress:  Selecting site:1 Stage in:1
>> Progress:  Submitting:1 Submitted:1
>> Progress:  Submitted:1 Failed but can retry:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/b/n/bgp000
>> Progress:  Submitted:1 Active:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/e/n/bgp000
>> Progress:  Submitted:1 Active:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/g/n/bgp000
>> Execution failed:
>>         Exception in echo:
>> Arguments: [Hello, world!]
>> Host: bgp000
>> Directory: first-20090317-1327-oqgttus8/jobs/g/n/echo-gnlq238j
>> stderr.txt:
>> 
>> stdout.txt:
>> 
>> ----
>> 
>> Caused by:
>>         Cannot transfer 
>> "/tmp/first-20090317-1327-oqgttus8/shared/hello.txt" to 
>> "/gpfs/home/zzhang/new_dock6/./hello.txt"
>> Caused by:
>>         No such file
>> 
>> 
>> Mihael Hategan wrote:
>> > On Tue, 2009-03-17 at 12:23 -0500, Zhao Zhang wrote:
>> >   
>> >> Hi, Mihael
>> >>
>> >> yes, can I do that?
>> >>     
>> >
>> > You should know this by now:
>> > in vdl-int.k, in doStageout, comment out the
task:transfer invocation
>> > (and dir:make).
>> >
>> >   
>> >> zhao
>> >>
>> >> Mihael Hategan wrote:
>> >>     
>> >>> On Tue, 2009-03-17 at 12:14 -0500, Zhao Zhang wrote:
>> >>>   
>> >>>       
>> >>>> Here comes another question, is there any place that I
could set to 
>> >>>> disable swift's waiting for data feature?
>> >>>>     
>> >>>>         
>> >>> Do you mean disable the stage-outs?
>> >>>
>> >>>   
>> >>>       
>> >>>> Or is there any way for me to cheat swift that the
data is already 
>> >>>> there? thanks.
>> >>>>
>> >>>> zhao
>> >>>>
>> >>>> Mihael Hategan wrote:
>> >>>>     
>> >>>>         
>> >>>>> On Fri, 2009-03-13 at 17:07 -0500, Zhao Zhang wrote:
>> >>>>>   
>> >>>>>       
>> >>>>>           
>> >>>>>> Hi, All
>> >>>>>>
>> >>>>>> I have a question on how swift knows if a task is
successful.
>> >>>>>> In my case, I am using a status notification instead
of a status file.
>> >>>>>>
>> >>>>>> So my question is is this status notification the
only thing swift is
>> >>>>>> waiting for, or is swift also waiting for the output
data to appear to
>> >>>>>> say that a job is successful?
>> >>>>>>     
>> >>>>>>         
>> >>>>>>             
>> >>>>> Once the job is done, swift will attempt to stage out
all the files that
>> >>>>> it expects the job to have produced.
>> >>>>>
>> >>>>> Should one of those files not be there, there will be
failures.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>   
>> >>>>>       
>> >>>>>           
>> >>>   
>> >>>       
>> >
>> >
>> >   
>
>_______________________________________________
>Swift-devel mailing list
>Swift-devel at ci.uchicago.edu
>http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel