[Swift-devel] How does swift know if a task is successful

[Zhao Zhang]

Hi, Sarah

skenny at uchicago.edu wrote:
> hey zhao, did you get this to work? 
Not yet, I am still working on it.
> was thinking i might try
> it on ranger, but i was wondering if you also then have to
> hack something else to prevent swift from cleaning up your
> work directory? 
I think to prevent swift cleaning up work dir is just an option in 
swift.properties.
> that is, i assume you actually DO want the
> output, you just don't want to have to wait for the stageouts (?)
>   
exactly, currently, we are building a collective IO system on BGP, so 
CIO will take
care of stage out results.

zhao
> ---- Original message ----
>   
>> Date: Tue, 17 Mar 2009 13:40:30 -0500
>> From: Mihael Hategan <hategan at mcs.anl.gov>  
>> Subject: Re: [Swift-devel] How does swift know if a task is
>>     
> successful  
>   
>> To: Zhao Zhang <zhaozhang at uchicago.edu>
>> Cc: swift-devel <swift-devel at ci.uchicago.edu>
>>
>> On Tue, 2009-03-17 at 13:36 -0500, Zhao Zhang wrote:
>>     
>>> Hi, Mihael
>>>
>>> I commented the following lines
>>>   /*dir:make(ldir)
>>>      restartOnError(".*", 2
>>>      task:transfer(srchost=host, srcfile=bname,
>>>      srcdir=rdir, destdir=ldir, desthost=dhost,
>>>       
> destprovider=provider)
>   
>>>   )*/
>>>
>>>       
>> Did you modify this file in dist/?/libexec? If not, did you
>>     
> re-compile
>   
>> swift after the modification?
>>
>> Put an echo or a log message in place, to see if your change
>>     
> is picked
>   
>> up by swift next time.
>>
>>     
>>> Then I modified wrapper.sh to not to copy output file back,
>>>       
> but I still 
>   
>>> got an error.
>>> The log file is at 
>>>
>>>       
> http://www.ci.uchicago.edu/~zzhang/first-20090317-1327-oqgttus8.log
>   
>>> Thanks
>>>
>>> zhao
>>>
>>> zzhang at login6.surveyor:~/new_dock6> ./run_swift_ssh.sh 1117
>>>       
> 64 first.swift
>   
>>> waiting for at least 64 nodes to register before submitting
>>>       
> workload...
>   
>>> waiting to find at least 1 services in file 
>>>
>>>       
> /home/falkon/users/zzhang/1117/config/Client-service-URIs.config...
>   
>>> all done, file has found at least 1 services
>>> found at least 64 registered, submitting workload...
>>> Swift svn swift-r2676 (swift modified locally) cog-r2305
>>>
>>> RunID: 20090317-1327-oqgttus8
>>> Progress:
>>> Progress:  Selecting site:1 Stage in:1
>>> Progress:  Submitting:1 Submitted:1
>>> Progress:  Submitted:1 Failed but can retry:1
>>> Failed to transfer wrapper log from 
>>> first-20090317-1327-oqgttus8/info/b/n/bgp000
>>> Progress:  Submitted:1 Active:1
>>> Failed to transfer wrapper log from 
>>> first-20090317-1327-oqgttus8/info/e/n/bgp000
>>> Progress:  Submitted:1 Active:1
>>> Failed to transfer wrapper log from 
>>> first-20090317-1327-oqgttus8/info/g/n/bgp000
>>> Execution failed:
>>>         Exception in echo:
>>> Arguments: [Hello, world!]
>>> Host: bgp000
>>> Directory: first-20090317-1327-oqgttus8/jobs/g/n/echo-gnlq238j
>>> stderr.txt:
>>>
>>> stdout.txt:
>>>
>>> ----
>>>
>>> Caused by:
>>>         Cannot transfer 
>>> "/tmp/first-20090317-1327-oqgttus8/shared/hello.txt" to 
>>> "/gpfs/home/zzhang/new_dock6/./hello.txt"
>>> Caused by:
>>>         No such file
>>>
>>>
>>> Mihael Hategan wrote:
>>>       
>>>> On Tue, 2009-03-17 at 12:23 -0500, Zhao Zhang wrote:
>>>>   
>>>>         
>>>>> Hi, Mihael
>>>>>
>>>>> yes, can I do that?
>>>>>     
>>>>>           
>>>> You should know this by now:
>>>> in vdl-int.k, in doStageout, comment out the
>>>>         
> task:transfer invocation
>   
>>>> (and dir:make).
>>>>
>>>>   
>>>>         
>>>>> zhao
>>>>>
>>>>> Mihael Hategan wrote:
>>>>>     
>>>>>           
>>>>>> On Tue, 2009-03-17 at 12:14 -0500, Zhao Zhang wrote:
>>>>>>   
>>>>>>       
>>>>>>             
>>>>>>> Here comes another question, is there any place that I
>>>>>>>               
> could set to 
>   
>>>>>>> disable swift's waiting for data feature?
>>>>>>>     
>>>>>>>         
>>>>>>>               
>>>>>> Do you mean disable the stage-outs?
>>>>>>
>>>>>>   
>>>>>>       
>>>>>>             
>>>>>>> Or is there any way for me to cheat swift that the
>>>>>>>               
> data is already 
>   
>>>>>>> there? thanks.
>>>>>>>
>>>>>>> zhao
>>>>>>>
>>>>>>> Mihael Hategan wrote:
>>>>>>>     
>>>>>>>         
>>>>>>>               
>>>>>>>> On Fri, 2009-03-13 at 17:07 -0500, Zhao Zhang wrote:
>>>>>>>>   
>>>>>>>>       
>>>>>>>>           
>>>>>>>>                 
>>>>>>>>> Hi, All
>>>>>>>>>
>>>>>>>>> I have a question on how swift knows if a task is
>>>>>>>>>                   
> successful.
>   
>>>>>>>>> In my case, I am using a status notification instead
>>>>>>>>>                   
> of a status file.
>   
>>>>>>>>> So my question is is this status notification the
>>>>>>>>>                   
> only thing swift is
>   
>>>>>>>>> waiting for, or is swift also waiting for the output
>>>>>>>>>                   
> data to appear to
>   
>>>>>>>>> say that a job is successful?
>>>>>>>>>     
>>>>>>>>>         
>>>>>>>>>             
>>>>>>>>>                   
>>>>>>>> Once the job is done, swift will attempt to stage out
>>>>>>>>                 
> all the files that
>   
>>>>>>>> it expects the job to have produced.
>>>>>>>>
>>>>>>>> Should one of those files not be there, there will be
>>>>>>>>                 
> failures.
>   
>>>>>>>>
>>>>>>>>   
>>>>>>>>       
>>>>>>>>           
>>>>>>>>                 
>>>>>>   
>>>>>>       
>>>>>>             
>>>>   
>>>>         
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>     
>
>