[Swift-devel] How does swift know if a task is successful

[Zhao Zhang]

Hi, Ben

Ben Clifford wrote:
> On Tue, 17 Mar 2009, Zhao Zhang wrote:
>
>   
>> exactly, currently, we are building a collective IO system on BGP, so CIO will
>> take
>> care of stage out results.
>>     
>
> Is it possible to interface between Swift and your work more cleanly?
>
> (maybe, for example, by doing something with the cog file transfer 
> provider API)
>   
Yes, make a new provider for swift is another way to do this.
> Hacking essential pieces of the Swift code out feels really unpleasant, 
> and will pretty much definitely break some functionality, which will cause 
> you trouble later on if you try to run arbitrary SwiftScript programs.
>   
Well, I agree with your point for production use. But things we are 
doing now is a research for a better
architecture  of swift on BGP.
> When we sat down in November/December, it sounded like you wouldn't need 
> to do anything like this to make the CIO stuff work with Swift; 
As I implemented what we discussed last time, new problem came up. 
Considering a 2-stage computation,
the second stage would take the output from the first as an input. With 
either ssh provider or gridftp provider,
this intermediate data has to be copied back to GPFS since the job that 
consumes this data could be sent to
any "site (pset)".

To solve this problem, we built a P2P data network on BGP over torus 
network. So the basic logic for this is
that if a wrapper.sh found a piece of intermediate data, it registered 
this data with (name, rank of the CN) to a
Centralized Hash Table(CHT). Next time, when a job needs this data, 
first it looks this data up in CHT, gets
the rank of the remote node, convert the RANK to IP, fetch the data 
directly.

With the above solution, all intermediate data has not to be copied back 
to GPFS, but swift are waiting for those
intermediate data to determine if the jobs of first stage are successful 
or not. In this case, swift won't send out the
jobs of 2nd stage, that's why we need to disable swift's staging out, 
and let swift determine a job status only by
provider status notification.

zhao
> so I'd be 
> interested in more explanation/discussion about what the CIO work looks 
> like now.
>
>