[Swift-devel] Concurrent dostagein calls limited to 8 ?

[Michael Wilde]

That explains a lot - the limited number of Karajan threads probably explains why coasters goes haywire in the larger tests as well.

Clearly this should be done as full fledged provider.  But that will take a fair bit more work.

Would there be any ill effects from bumping up the number of karajan threads to see if I can make this demo work?  WHere is that set?

Also, when you say "use the local provider or
> some other scheme that can free the workers while the sub-processes
> run." - do you have anything "quick and easy" in mind there?

- Mike


----- Original Message -----
> The cdm functions (externalin, externalout, externalgo) are not
> asynchronous. They block the karajan worker threads and therefore,
> besides preventing anything else from running in the interpreter, are
> also limited to concurrently running whatever the number of karajan
> worker threads is (2*cores).
> 
> I would suggest changing those functions to use the local provider or
> some other scheme that can free the workers while the sub-processes
> run.
> 
> Mihael
> 
> On Sun, 2010-11-14 at 20:56 -0600, Michael Wilde wrote:
> > I'm in a cab - vdlint.k is in local fs on:
> >
> > Login1.pads.ci
> > /scratch/local/wilde/swift/src/trunk/...
> > Running from dist/swft-svn in that tree
> >
> > On 11/14/10, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > > On Sun, 2010-11-14 at 17:23 -0600, Michael Wilde wrote:
> > >> Some answers from my handheld:
> > >> - foreach loop has 317 files so ample parallelism
> > >
> > > I would have assumed it's > 8. But I suspect, given one of the
> > > answers
> > > below, that it does not matter.
> > >
> > >> - throttle in sites entry set to .63 to run 64 jobs at once
> > >> - the "active" external.sh is called from end of dostagein and
> > >> dostageout in vdl-int.k (after all files for the job were put in
> > >> a
> > >> list by prior calls to externa.sh from within those functions
> > >
> > > How is this call actually implemented. I.e. can you post the
> > > respective
> > > snippet of vdl-int?
> > >
> > >> - the actual staging op by globusonline take 30-60 seconds,
> > >> sometimes
> > >> more. I batch them up.
> > >
> > >
> > >
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory