[Swift-devel] GPFS issue of SWIFT on BGP

[Mihael Hategan]

----- Zhao Zhang <zhaozhang at uchicago.edu> wrote:
> Hi, All
> 
> I am working with Allan on applying CIO to SWIFT on BGP, now we are 
> blocked by a ssh-provider issue.
> Here is the description: we made ssh-provider working as the data 
> provider, and I tested it with multiple psets, it is working fine.
> Login Node ----- submit host
> IO Node -------- remote site
> Compute Node -- workers
> 
> Now, we start swift on Login Node, and the working directory will be 
> created on IO Node, so that all intermediate files and final
> result files will be copied back to Login Node(GPFS) once they are 
> generated. Here we got an old problem, all IO nodes are trying
> to write files in the same directory, which we are trying to avoid all 
> the way.
> 
> My solution would be modify the ssh-provider source code, implement an 
> asynchronous collector logic there.

I'm not really sure what the protocol used to move data with has to do with
organizing files to make things work faster.