[Swift-user] throttle transfers and vdl:stagein graphs

[Allan Espinosa]

Ah.  I see that the the log entries that is nearest the actual
task:transfer() task is in vdl:dostageinfile (no parallelFor loop
here).  But i still see the transfers being more than the throttle in
swift.properties.

One of the classes invoked by task:transfer() is
/org/globus/cog/karajan/workflow/nodes/grid/GridTransfer.class ?

I'll try adding this to my log4j.properties and see what will happen.

-Allan


2010/11/8 Mihael Hategan <hategan at mcs.anl.gov>:
> On Mon, 2010-11-08 at 20:50 -0600, Allan Espinosa wrote:
>> Hi,
>>
>> In my workflow, I use the default throttle.transfers=4 .  But my
>> dostagein-total plot indicates that there are 72 stagein events going
>> on for around 90 seconds.  shouldn't there be a linear ramp up or a
>> saw-tooth pattern at the plateau because of having throttled
>> transfers?
>
> Lies. And statistics.
>
> The plot indicates that a number of instances of a certain portion of
> vdl-int is executing.
>
> If you look at that portion of vdl-int (i.e. between setprogress("Stage
> in") and setprogress("Submitting")) there are a few things happening,
> including directory creation.
>
> Essentially you are dealing with the following pattern:
>
> parallelFor(...
>  a()
>  throttle(4, b())
>  c()
> )
>
> The graph would show something like the parallelism in the invocation of
> the body of parallelFor. And it is quite possible that all a()
> invocations start well before any of the b() invocations start. The only
> accurate way to see the effect of the throttle is to trace the b()
> invocations, which you can probably do by looking at the status of file
> transfer tasks (by enabling the relevant logging stuff).
>
> Mihael