[Swift-devel] Concurrent dostagein calls limited to 8 ?

[Michael Wilde]

Mihael,

I attached my vdl-int.k.  The changes were based on Justin's initial version of the external policy CDM setting, but I added the ability to handle stageout as well as stagein, and to gather all the files for a stagein or stageout in the external script, and process them all at once.

In my external script, I now batch the files for multiple requests into one larger transfer command to globusonline, using time-based batching.  This adds latency to an individual request but saves greatly overall, as globusonline will only do 3 concurrent transfers for a given user, and has its own latency for checking its work queue.

My hooks are the calls to cdm:externalin, externalout, and externalgo.

I use a map element as a reference variable to determine when to call externalgo.

All this seems to work at the basic level, but I still see only a steady state of 8 external calls running at once.

Further, I think the latency involved is causing some strange interaction with coasters which I need to send you.  My scripts run fine on localhost but fail on PADS with coasters: after about 80 of 300+ jobs I get a caster failure that I need to log and post - looks like some kind of timeout in worker.pl waiting for a response.

- Mike


----- Original Message -----
> On Sun, 2010-11-14 at 17:23 -0600, Michael Wilde wrote:
> > Some answers from my handheld:
> > - foreach loop has 317 files so ample parallelism
> 
> I would have assumed it's > 8. But I suspect, given one of the answers
> below, that it does not matter.
> 
> > - throttle in sites entry set to .63 to run 64 jobs at once
> > - the "active" external.sh is called from end of dostagein and
> > dostageout in vdl-int.k (after all files for the job were put in a
> > list by prior calls to externa.sh from within those functions
> 
> How is this call actually implemented. I.e. can you post the
> respective
> snippet of vdl-int?
> 
> > - the actual staging op by globusonline take 30-60 seconds,
> > sometimes
> > more. I batch them up.

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdl-int.k
Type: application/octet-stream
Size: 20559 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20101114/d8017301/attachment.obj>