[Swift-devel] Re: Provider staging error in long-running test

[Mihael Hategan]

Ok. So that doesn't look like it's a staging problem specifically, but
more like something with the comm library. I'll have to look at the
logs. And I can foresee some free time coming in a couple of days just
for that!

Mihael

On Sun, 2010-11-21 at 23:10 -0600, Michael Wilde wrote:
> Mihael, here is bug 3:
> 
> I was testing a foreach loop doing a cat of 10,000 input files of sizes up to about 300-400K each.  The test hit an error after around 3,491 files:
> 
> Progress:  Selecting site:1008  Submitted:12  Active:3  Finished successfully:3476
> Progress:  Selecting site:1008  Submitted:13  Active:3  Finished successfully:3491
> Failed to shut down channel
> java.lang.NullPointerException
>         at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.configureHeartBeat(AbstractKarajanChannel.java:57)
>         at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.<init>(AbstractKarajanChannel.java:52)
> 
> The test was executed on PADS login1 like this:
> 
> cd /home/wilde/swift/lab
> ./run.local.coast.ps.sh catsall
> 
> log file: catsall-20101121-2239-oc2flmn0.log
> 
> sites.xml:
> 
> <config>
>   <pool handle="localhost">
>     <!-- <execution provider="coaster-persistent" url="http://login1.pads.ci.uchicago.edu:" jobmanager="local:local"/> -->
>     <execution provider="coaster" url="none" jobmanager="local:local"/>
>     <!-- <profile namespace="globus" key="workerManager">passive</profile> -->
>     <profile namespace="globus" key="workersPerNode">8</profile>
>     <profile namespace="globus" key="slots">1</profile>
>     <profile namespace="globus" key="maxnodes">1</profile>
>     <profile key="jobThrottle" namespace="karajan">.15</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
>     <profile namespace="swift" key="stagingMethod">proxy</profile>
>     <workdirectory>/scratch/local/wilde/pstest/swiftwork</workdirectory>
>   </pool>
> </config>
> 
> login1$ cat cf
> wrapperlog.always.transfer=true
> sitedir.keep=true
> execution.retries=0
> lazy.errors=false
> status.mode=provider
> use.provider.staging=true
> provider.staging.pin.swiftfiles=false
> login1$ cat catsall.swift
> type file;
> 
> app (file o) cat (file i)
> {
>   cat @i stdout=@o;
> }
> 
> file infile[]  <simple_mapper; location="indir", prefix="f.", suffix=".in">;
> file outfile[] <simple_mapper; location="outdir", prefix="f.",suffix=".out">;
> 
> foreach f, i in infile {
>   outfile[i] = cat(f);
> }
> login1$ 
> 
> login1$ which swift
> /scratch/local/wilde/swift/src/trunk.gomods/cog/modules/swift/dist/swift-svn/bin/swift
> login1$ java -version
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> login1$ 
> 
>