[Swift-devel] coaster transfer performance

[Michael Wilde]

Im moving this discussion to swift-devel.

Mihael's fix to the timeout problem works very well - that problem has not recurred so far as I can tell.

We then discussed performance improvements for coaster provider staging.

Mihael suggested this:

"there should be a commented-out line in GSSChannel that says gssContext.requestConf(false); (line 85)
uncomment that and re-compile, it will disable encryption"

...and observed this:

> Disabling encryption makes the stage-ins go from about 6 MB/s to 70-90 MB/s.
> 
> However, stageouts are still slow (about 4 MB/s).

I tried that, and it speeds things up a great deal. (No measurements from the fMRI demo case yet; measurements from Mihael below.)

I also tried increasing the transfer buffer size. The current buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).

At 512KB, staging-in to the app goes "very fast" (again, no measures yet) but output hangs after about 25 of 100 transfers.  At 128KB, it hasn't hung yet, and goes very fast.  I too am seeing slower stage-out times from the app back to swift than stage=in times.

I'm testing on midway's shared fs right now (GPFS?) and will test on hard disk next.

But with these two fixes, things are working with very nice reliability and performance in tests so far.

The mods for these changes are below, followed by Mihael's report on testing w/o encryption.

Yadu, please integrate your coaster-provider-staging tests into the test suite and test across a range of file sizes, durations, and endpoints. Im happy to discuss this, on this list and/or in person.

Thanks,

- Mike

mid$ cat mods
modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
modules/provider-coaster/resources/worker.pl
modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
mid$ svn diff
Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
===================================================================
--- modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (revision 3672)
+++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (working copy)
@@ -24,7 +24,7 @@
 public class Buffers extends Thread {
     public static final Logger logger = Logger.getLogger(Buffers.class);
 
-    public static final int ENTRY_SIZE = 32768;
+  public static final int ENTRY_SIZE = 32768 * 4;
     public static final int ENTRIES_PER_STREAM = 8;
     public static final int MAX_ENTRIES = 1024; // 32 MB
     public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
Index: modules/provider-coaster/resources/worker.pl
===================================================================
--- modules/provider-coaster/resources/worker.pl        (revision 3672)
+++ modules/provider-coaster/resources/worker.pl        (working copy)
@@ -134,7 +134,7 @@
 my $JOB_COUNT = 0;
 
 use constant BUFSZ => 2048;
-use constant IOBUFSZ => 32768;
+use constant IOBUFSZ => 32768 * 4;
 use constant IOBLOCKSZ => 8;
 
 # If true, enable a profile result that is written to the log
Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
===================================================================
--- modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (revision 3672)
+++ modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (working copy)
@@ -82,7 +82,7 @@
 
                                gssContext.requestAnonymity(false);
                                gssContext.requestCredDeleg(false);
-                               //gssContext.requestConf(false);
+                               gssContext.requestConf(false); // Uncommented to disable encryption
                                gssContext.setOption(GSSConstants.GSS_MODE, GSIConstants.MODE_SSL);
                                gssContext.setOption(GSSConstants.DELEGATION_TYPE,
                                                GSIConstants.DELEGATION_TYPE_LIMITED);
mid$ 



----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Sent: Thursday, June 13, 2013 10:47:18 PM
> Subject: coaster transfer performance
> 
> Hi,
> 
> This is what I'm seeing. Disabling encryption makes the stage-ins go
> from about 6 MB/s to 70-90 MB/s.
> 
> However, stageouts are still slow (about 4 MB/s). It turns out that
> this
> is not due to the shared FS. I disabled writing to disk completely,
> and
> the performance is still around 6 MB/s.
> 
> I'll need to find out why, but probably not tonight.
> 
> Mihael
> 
>