[Swift-devel] How does swift know if a task is successful

[Zhao Zhang]

Hi, Ben

Ben Clifford wrote:
> So if Swift could remove the dependency between staging out and starting 
> subsequent jobs (a subset of what has been talked about before), would you 
> still need to hack out the stageout code?
>   
I think swift still needs to hold the 2nd stage computation until the 
1st completes. If we simply remove
the dependency, swift would send all jobs (both 1st and 2nd) out, right?
>   
>> To solve this problem, we built a P2P data network on BGP over torus 
>> network. So the basic logic for this is that if a wrapper.sh found a 
>> piece of intermediate data, it registered this data with (name, rank of 
>> the CN) to a Centralized Hash Table(CHT). Next time, when a job needs 
>> this data, first it looks this data up in CHT, gets the rank of the 
>> remote node, convert the RANK to IP, fetch the data directly.
>>     
>
> When we talked in December, I think this bit was done with posix 
> filesystem access. 
We missed this point in last talk.
> But it sounds like you are doing something different 
> now.
>
> I've looked at abstracting that worker<->site shared filesystem code in 
> the past (and have some patches floating round in half-written state) - 
> can you send me your modified wrapper.sh so I can see how you do things?
>   
Here it is: http://www.ci.uchicago.edu/~zzhang/wrapper.sh

zhao