[Swift-user] throttle transfers and vdl:stagein graphs

[Mihael Hategan]

On Mon, 2010-11-08 at 20:50 -0600, Allan Espinosa wrote:
> Hi,
> 
> In my workflow, I use the default throttle.transfers=4 .  But my
> dostagein-total plot indicates that there are 72 stagein events going
> on for around 90 seconds.  shouldn't there be a linear ramp up or a
> saw-tooth pattern at the plateau because of having throttled
> transfers?

Lies. And statistics.

The plot indicates that a number of instances of a certain portion of
vdl-int is executing.

If you look at that portion of vdl-int (i.e. between setprogress("Stage
in") and setprogress("Submitting")) there are a few things happening,
including directory creation.

Essentially you are dealing with the following pattern:

parallelFor(...
  a()
  throttle(4, b())
  c()
)

The graph would show something like the parallelism in the invocation of
the body of parallelFor. And it is quite possible that all a()
invocations start well before any of the b() invocations start. The only
accurate way to see the effect of the throttle is to trace the b()
invocations, which you can probably do by looking at the status of file
transfer tasks (by enabling the relevant logging stuff).

Mihael