[Swift-devel] coaster transfer performance

[Mihael Hategan]

I would like to point out that I have not seen a difference on stage-out
performance when adjusting the buffer size (on the worker side). I have
*not* tried a similar experiment for the stage-in buffer size, and I
believe that before we increase that permanently, we should see if it
does indeed make a difference.

Mihael

On Fri, 2013-06-14 at 08:19 -0500, Michael Wilde wrote:
> Im moving this discussion to swift-devel.
> 
> Mihael's fix to the timeout problem works very well - that problem has not recurred so far as I can tell.
> 
> We then discussed performance improvements for coaster provider staging.
> 
> Mihael suggested this:
> 
> "there should be a commented-out line in GSSChannel that says gssContext.requestConf(false); (line 85)
> uncomment that and re-compile, it will disable encryption"
> 
> ...and observed this:
> 
> > Disabling encryption makes the stage-ins go from about 6 MB/s to 70-90 MB/s.
> > 
> > However, stageouts are still slow (about 4 MB/s).
> 
> I tried that, and it speeds things up a great deal. (No measurements from the fMRI demo case yet; measurements from Mihael below.)
> 
> I also tried increasing the transfer buffer size. The current buffer is 32KB.  I tried 16X (512KB) and 4X (128KB).
> 
> At 512KB, staging-in to the app goes "very fast" (again, no measures yet) but output hangs after about 25 of 100 transfers.  At 128KB, it hasn't hung yet, and goes very fast.  I too am seeing slower stage-out times from the app back to swift than stage=in times.
> 
> I'm testing on midway's shared fs right now (GPFS?) and will test on hard disk next.
> 
> But with these two fixes, things are working with very nice reliability and performance in tests so far.
> 
> The mods for these changes are below, followed by Mihael's report on testing w/o encryption.
> 
> Yadu, please integrate your coaster-provider-staging tests into the test suite and test across a range of file sizes, durations, and endpoints. Im happy to discuss this, on this list and/or in person.
> 
> Thanks,
> 
> - Mike
> 
> mid$ cat mods
> modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> modules/provider-coaster/resources/worker.pl
> modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> mid$ svn diff
> Index: modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java
> ===================================================================
> --- modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (revision 3672)
> +++ modules/provider-coaster/src/org/globus/cog/abstraction/impl/file/coaster/buffers/Buffers.java      (working copy)
> @@ -24,7 +24,7 @@
>  public class Buffers extends Thread {
>      public static final Logger logger = Logger.getLogger(Buffers.class);
>  
> -    public static final int ENTRY_SIZE = 32768;
> +  public static final int ENTRY_SIZE = 32768 * 4;
>      public static final int ENTRIES_PER_STREAM = 8;
>      public static final int MAX_ENTRIES = 1024; // 32 MB
>      public static final int PERFORMANCE_LOGGING_INTERVAL = 10000;
> Index: modules/provider-coaster/resources/worker.pl
> ===================================================================
> --- modules/provider-coaster/resources/worker.pl        (revision 3672)
> +++ modules/provider-coaster/resources/worker.pl        (working copy)
> @@ -134,7 +134,7 @@
>  my $JOB_COUNT = 0;
>  
>  use constant BUFSZ => 2048;
> -use constant IOBUFSZ => 32768;
> +use constant IOBUFSZ => 32768 * 4;
>  use constant IOBLOCKSZ => 8;
>  
>  # If true, enable a profile result that is written to the log
> Index: modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java
> ===================================================================
> --- modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (revision 3672)
> +++ modules/karajan/src/org/globus/cog/karajan/workflow/service/channels/GSSChannel.java        (working copy)
> @@ -82,7 +82,7 @@
>  
>                                 gssContext.requestAnonymity(false);
>                                 gssContext.requestCredDeleg(false);
> -                               //gssContext.requestConf(false);
> +                               gssContext.requestConf(false); // Uncommented to disable encryption
>                                 gssContext.setOption(GSSConstants.GSS_MODE, GSIConstants.MODE_SSL);
>                                 gssContext.setOption(GSSConstants.DELEGATION_TYPE,
>                                                 GSIConstants.DELEGATION_TYPE_LIMITED);
> mid$ 
> 
> 
> 
> ----- Original Message -----
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "Michael Wilde" <wilde at mcs.anl.gov>
> > Sent: Thursday, June 13, 2013 10:47:18 PM
> > Subject: coaster transfer performance
> > 
> > Hi,
> > 
> > This is what I'm seeing. Disabling encryption makes the stage-ins go
> > from about 6 MB/s to 70-90 MB/s.
> > 
> > However, stageouts are still slow (about 4 MB/s). It turns out that
> > this
> > is not due to the shared FS. I disabled writing to disk completely,
> > and
> > the performance is still around 6 MB/s.
> > 
> > I'll need to find out why, but probably not tonight.
> > 
> > Mihael
> > 
> >