[Swift-devel] Next provider staging enhancements

[Michael Wilde]

Mihael, Justin, and I spoke by phone on Friday on this topic. The outcome was that Mihael will work on the following provider staging enhancements:

1. Make non-encrypted data transfer the default for provider staging. This yields 20X performance improvement (transfer speed) in laptop tests. 

One suspected problem: Ketan tried testing this by manually changing the Java code flag mentioned in Mihael's email on the topic, and did not observe any speedup. So that needs to be investigated: was the setting incorrect, or did it not yield the expected improvement?  We should set up a test that demonstrates the expected speed range, and make that speed test a part of the test suite.

2. Mihael will test and if necessary fix the ability of provider staging to work with gsiftp:// URIs.  Ketan will create a test for this as well, and then adapt the ExTENCI applications to use this method.

3. Mihael will adapt the provider staging "SFS" (shared filesystem) staging method to create a new staging method (like worker-gridftp) which uses globus-url-copy on the worker node as the staging mechanism. That could perhaps be further generalized, but that generalization could come later. Justin pointed out that in order to not have the worker block other job execution activities during these staging operations, the globus-url-copy processes need to be made asynchronous to normal worker job processing.

Mihael proposed, I think, to do item 1 right away, for 0.93, and to do items 2 & 3 in a new 0.93.1 branch.  That way they will not slow down release of 0.93, but will be made available much sooner than they would be if they were made on trunk and then slotted for 0.94.

Mihael, all, can you comment on whether I have described this all right?

Thanks,

- Mike