[Swift-devel] Provider staging error in long-running test

[Michael Wilde]

Mihael, here is bug 3:

I was testing a foreach loop doing a cat of 10,000 input files of sizes up to about 300-400K each.  The test hit an error after around 3,491 files:

Progress:  Selecting site:1008  Submitted:12  Active:3  Finished successfully:3476
Progress:  Selecting site:1008  Submitted:13  Active:3  Finished successfully:3491
Failed to shut down channel
java.lang.NullPointerException
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.configureHeartBeat(AbstractKarajanChannel.java:57)
        at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.<init>(AbstractKarajanChannel.java:52)

The test was executed on PADS login1 like this:

cd /home/wilde/swift/lab
./run.local.coast.ps.sh catsall

log file: catsall-20101121-2239-oc2flmn0.log

sites.xml:

<config>
  <pool handle="localhost">
    <!-- <execution provider="coaster-persistent" url="http://login1.pads.ci.uchicago.edu:" jobmanager="local:local"/> -->
    <execution provider="coaster" url="none" jobmanager="local:local"/>
    <!-- <profile namespace="globus" key="workerManager">passive</profile> -->
    <profile namespace="globus" key="workersPerNode">8</profile>
    <profile namespace="globus" key="slots">1</profile>
    <profile namespace="globus" key="maxnodes">1</profile>
    <profile key="jobThrottle" namespace="karajan">.15</profile>
    <profile namespace="karajan" key="initialScore">10000</profile>
    <profile namespace="swift" key="stagingMethod">proxy</profile>
    <workdirectory>/scratch/local/wilde/pstest/swiftwork</workdirectory>
  </pool>
</config>

login1$ cat cf
wrapperlog.always.transfer=true
sitedir.keep=true
execution.retries=0
lazy.errors=false
status.mode=provider
use.provider.staging=true
provider.staging.pin.swiftfiles=false
login1$ cat catsall.swift
type file;

app (file o) cat (file i)
{
  cat @i stdout=@o;
}

file infile[]  <simple_mapper; location="indir", prefix="f.", suffix=".in">;
file outfile[] <simple_mapper; location="outdir", prefix="f.",suffix=".out">;

foreach f, i in infile {
  outfile[i] = cat(f);
}
login1$ 

login1$ which swift
/scratch/local/wilde/swift/src/trunk.gomods/cog/modules/swift/dist/swift-svn/bin/swift
login1$ java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
login1$ 


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory