[Swift-user] throttle transfers and vdl:stagein graphs

[Allan Espinosa]

I see this in doStagein:

			uParallelFor(file, files
				provider := provider(file)
				srchost := hostname(file)
				srcdir := vdl:dirname(file)
				destdir := dircat(dir, reldirname(file))
				filename := basename(file)
				size := file:size("{srcdir}/{filename}", host=srchost, provider=provider)

				policy := cdm:query(query=file)
				log(LOG:DEBUG, "CDM: {file} : {policy}")

				doStageinFile(provider=provider, srchost=srchost, srcfile=filename,
						srcdir=srcdir, desthost=host, destdir=destdir, size=size, policy=policy)
			)
			log(LOG:INFO, "END jobid={jobid} - Staging in finished")

Does this mean that there is actually no throttling going on for
dostageinfile() ?   It does make sense since my 400k-job workflow is
still stuck for 5 hours staging in 23k files.

-Allan


2010/11/8 Mihael Hategan <hategan at mcs.anl.gov>:
> On Mon, 2010-11-08 at 20:50 -0600, Allan Espinosa wrote:
>> Hi,
>>
>> In my workflow, I use the default throttle.transfers=4 .  But my
>> dostagein-total plot indicates that there are 72 stagein events going
>> on for around 90 seconds.  shouldn't there be a linear ramp up or a
>> saw-tooth pattern at the plateau because of having throttled
>> transfers?
>
> Lies. And statistics.
>
> The plot indicates that a number of instances of a certain portion of
> vdl-int is executing.
>
> If you look at that portion of vdl-int (i.e. between setprogress("Stage
> in") and setprogress("Submitting")) there are a few things happening,
> including directory creation.
>
> Essentially you are dealing with the following pattern:
>
> parallelFor(...
>  a()
>  throttle(4, b())
>  c()
> )
>
> The graph would show something like the parallelism in the invocation of
> the body of parallelFor. And it is quite possible that all a()
> invocations start well before any of the b() invocations start. The only
> accurate way to see the effect of the throttle is to trace the b()
> invocations, which you can probably do by looking at the status of file
> transfer tasks (by enabling the relevant logging stuff).
>
> Mihael