[Swift-devel] Re: Concurrent dostagein calls limited to 8 ?

[Michael Wilde]

As far as I can tell its still showing the same behavior - only 8 stageins are active:

wilde    17568 17564 17564  4873  0 16:05 pts/8    00:00:00             /bin/sh /scratch/local/wilde/swift/src/trunk/cog/modules/swift
wilde    17632 17568 17564  4873  5 16:05 pts/8    00:00:09               java -Xmx256M -Djava.endorsed.dirs=/scratch/local/wilde/swif
wilde    19193 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19357 19193 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19198 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19358 19198 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19205 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19359 19205 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19210 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19361 19210 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19214 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19362 19214 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19218 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19363 19218 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19224 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19364 19224 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3
wilde    19240 17632 17564  4873  0 16:08 pts/8    00:00:00                 /bin/bash /home/wilde/swift/lab/go/external.sh getlanduse-
wilde    19365 19240 17564  4873  0 16:08 pts/8    00:00:00                   sleep 3


- Mike


----- Original Message -----
> My bad. Try cog trunk r2932.
> 
> Mihael
> 
> On Sun, 2010-11-14 at 12:45 -0600, Michael Wilde wrote:
> > Im using Justin's new external CDM policy to talk to
> > globusonline.org.
> >
> > Ive got the two IO throttles set to 30:
> >
> > throttle.transfers=30
> > throttle.file.operations=30
> >
> > ...but I still see exactly 8 concurrent dostagein calls (as seen by
> > calls to my external IO handler external.sh)
> >
> > Do you know what is limiting this concurrency to 8? I'd like to open
> > it up much wider.
> >
> > My swift command and relevant files are below.
> > Logs are in CI: /home/wilde/swift/demo/modis
> >
> > Thanks,
> >
> > Mike
> >
> >
> > login1$ cat rundemo.go.local.sh
> > rm -f external.*.log
> > swift -config cf -tc.file tc.local -sites.file sites.local.xml
> > -cdm.file fs.ftponly modis.go.swift -nfiles=30
> > # -location= -n= -site==
> >
> > login1$ cat cf
> > wrapperlog.always.transfer=true
> > sitedir.keep=true
> > execution.retries=0
> > lazy.errors=false
> > status.mode=provider
> > use.provider.staging=false
> > provider.staging.pin.swiftfiles=false
> > throttle.transfers=30
> > throttle.file.operations=30
> >
> > login1$ cat sites.local.xml
> > <config>
> >
> >   <pool handle="localhost">
> >     <execution provider="local" url="none"/>
> >     <profile namespace="karajan" key="jobThrottle">.63</profile>
> >     <profile namespace="karajan" key="initialScore">10000</profile>
> >     <filesystem provider="local"/>
> >     <workdirectory>/home/wilde/swiftwork</workdirectory>
> >   </pool>
> >
> > </config>
> > login1$
> >
> > login1$ cat fs.ftponly
> > rule .*gsiftp://.* EXTERNAL /home/wilde/swift/lab/go/external.sh
> > login1$
> >
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory