[Swift-devel] Concurrent dostagein calls limited to 8 ?

[Mihael Hategan]

The cdm functions (externalin, externalout, externalgo) are not
asynchronous. They block the karajan worker threads and therefore,
besides preventing anything else from running in the interpreter, are
also limited to concurrently running whatever the number of karajan
worker threads is (2*cores).

I would suggest changing those functions to use the local provider or
some other scheme that can free the workers while the sub-processes run.

Mihael

On Sun, 2010-11-14 at 20:56 -0600, Michael Wilde wrote:
> I'm in a cab - vdlint.k is in local fs on:
> 
> Login1.pads.ci
> /scratch/local/wilde/swift/src/trunk/...
> Running from dist/swft-svn in that tree
> 
> On 11/14/10, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > On Sun, 2010-11-14 at 17:23 -0600, Michael Wilde wrote:
> >> Some answers from my handheld:
> >> - foreach loop has 317 files so ample parallelism
> >
> > I would have assumed it's > 8. But I suspect, given one of the answers
> > below, that it does not matter.
> >
> >> - throttle in sites entry set to .63 to run 64 jobs at once
> >> - the "active" external.sh is called from end of dostagein and
> >> dostageout in vdl-int.k (after all files for the job were put in a
> >> list by prior calls to externa.sh from within those functions
> >
> > How is this call actually implemented. I.e. can you post the respective
> > snippet of vdl-int?
> >
> >> - the actual staging op by globusonline take 30-60 seconds, sometimes
> >> more. I batch them up.
> >
> >
> >
>