[Swift-devel] How does swift know if a task is successful

[Zhao Zhang]

Hi, Mihael

I modified the vdl-int.k in cog/module/swift/libexec, and rebuilt swift, 
and I used my customized wrapper.sh.
I ran the first.swift as a test,  the job returned successful,  and the 
output file was still staged out. Any ideas?
Thanks.

zhao

Mihael Hategan wrote:
> On Tue, 2009-03-17 at 13:36 -0500, Zhao Zhang wrote:
>   
>> Hi, Mihael
>>
>> I commented the following lines
>>   /*dir:make(ldir)
>>      restartOnError(".*", 2
>>      task:transfer(srchost=host, srcfile=bname,
>>      srcdir=rdir, destdir=ldir, desthost=dhost, destprovider=provider)
>>   )*/
>>
>>     
>
> Did you modify this file in dist/?/libexec? If not, did you re-compile
> swift after the modification?
>
> Put an echo or a log message in place, to see if your change is picked
> up by swift next time.
>
>   
>> Then I modified wrapper.sh to not to copy output file back, but I still 
>> got an error.
>> The log file is at 
>> http://www.ci.uchicago.edu/~zzhang/first-20090317-1327-oqgttus8.log
>> Thanks
>>
>> zhao
>>
>> zzhang at login6.surveyor:~/new_dock6> ./run_swift_ssh.sh 1117 64 first.swift
>> waiting for at least 64 nodes to register before submitting workload...
>> waiting to find at least 1 services in file 
>> /home/falkon/users/zzhang/1117/config/Client-service-URIs.config...
>> all done, file has found at least 1 services
>> found at least 64 registered, submitting workload...
>> Swift svn swift-r2676 (swift modified locally) cog-r2305
>>
>> RunID: 20090317-1327-oqgttus8
>> Progress:
>> Progress:  Selecting site:1 Stage in:1
>> Progress:  Submitting:1 Submitted:1
>> Progress:  Submitted:1 Failed but can retry:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/b/n/bgp000
>> Progress:  Submitted:1 Active:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/e/n/bgp000
>> Progress:  Submitted:1 Active:1
>> Failed to transfer wrapper log from 
>> first-20090317-1327-oqgttus8/info/g/n/bgp000
>> Execution failed:
>>         Exception in echo:
>> Arguments: [Hello, world!]
>> Host: bgp000
>> Directory: first-20090317-1327-oqgttus8/jobs/g/n/echo-gnlq238j
>> stderr.txt:
>>
>> stdout.txt:
>>
>> ----
>>
>> Caused by:
>>         Cannot transfer 
>> "/tmp/first-20090317-1327-oqgttus8/shared/hello.txt" to 
>> "/gpfs/home/zzhang/new_dock6/./hello.txt"
>> Caused by:
>>         No such file
>>
>>
>> Mihael Hategan wrote:
>>     
>>> On Tue, 2009-03-17 at 12:23 -0500, Zhao Zhang wrote:
>>>   
>>>       
>>>> Hi, Mihael
>>>>
>>>> yes, can I do that?
>>>>     
>>>>         
>>> You should know this by now:
>>> in vdl-int.k, in doStageout, comment out the task:transfer invocation
>>> (and dir:make).
>>>
>>>   
>>>       
>>>> zhao
>>>>
>>>> Mihael Hategan wrote:
>>>>     
>>>>         
>>>>> On Tue, 2009-03-17 at 12:14 -0500, Zhao Zhang wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Here comes another question, is there any place that I could set to 
>>>>>> disable swift's waiting for data feature?
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> Do you mean disable the stage-outs?
>>>>>
>>>>>   
>>>>>       
>>>>>           
>>>>>> Or is there any way for me to cheat swift that the data is already 
>>>>>> there? thanks.
>>>>>>
>>>>>> zhao
>>>>>>
>>>>>> Mihael Hategan wrote:
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>> On Fri, 2009-03-13 at 17:07 -0500, Zhao Zhang wrote:
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>>>> Hi, All
>>>>>>>>
>>>>>>>> I have a question on how swift knows if a task is successful.
>>>>>>>> In my case, I am using a status notification instead of a status file.
>>>>>>>>
>>>>>>>> So my question is is this status notification the only thing swift is
>>>>>>>> waiting for, or is swift also waiting for the output data to appear to
>>>>>>>> say that a job is successful?
>>>>>>>>     
>>>>>>>>         
>>>>>>>>             
>>>>>>>>                 
>>>>>>> Once the job is done, swift will attempt to stage out all the files that
>>>>>>> it expects the job to have produced.
>>>>>>>
>>>>>>> Should one of those files not be there, there will be failures.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>   
>>>>>       
>>>>>           
>>>   
>>>       
>
>
>