[Swift-devel] running nightly.sh on pads
Sarah Kenny
skenny at uchicago.edu
Mon Jan 10 17:17:36 CST 2011
so, i'm trying to get nightly.sh to run on pads with coasters and i'm not
quite sure where this is falling apart. so far the only thing i've edited is
providers/ssh-pbs-coasters/sites.template.xml (allowing it to take the
PROJECT and QUEUE variables). from what i can tell the sites.xml file does
get generated correctly but then according to the test output it times out
during submission:
[skenny at login1 tests]$ ./nightly.sh -c -g -s groups/group-ssh.sh
RUNNING_IN:
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10
HTML_OUTPUT:
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10/tests-2011-01-10.html
which: no ifconfig in
(/ci/projects/cnari/apps/freesurfer64/bin:/ci/projects/cnari/apps/freesurfer64/fsfast/bin:/ci/projects/cnari/apps/freesurfer64/mni/bin:/ci/projects/cnari/usr/bin:/ci/projects/cnari/apps/afni:/ci/projects/cnari/apps/swift/bin:/soft/java-1.6.0_11-sun-r1/bin:/soft/java-1.6.0_11-sun-r1/jre/bin:/software/common/gx-map-0.5.3.3-r1/bin:/soft/apache-ant-1.7.1-r1/bin:/soft/condor-7.0.5-r1/bin:/soft/globus-4.2.1-r2/bin:/soft/globus-4.2.1-r2/sbin:/usr/kerberos/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/software/common/softenv-1.6.0-r1/bin:/home/skenny/bin/linux-rhel5-x86_64:/home/skenny/bin:/soft/maui-3.2.6p21-r1/bin:/soft/maui-3.2.6p21-r1/sbin:/soft/openmpi-1.4.2-gcc4.1-r1/bin)
GROUPLISTFILE: groups/group-ssh.sh
Prolog: Build
Executing (part 1)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
Executing (part 2)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
14815 pts/27 00:00:00 nightly.sh
monitor(1): killing test process...
touch: cannot touch `killed_test': Stale NFS file handle
monitor(1): killed process_exec (TERM)
process_exec_trap()
killing all swifts...
++ echo 13685
13685
++ ps -f
UID PID PPID C STIME TTY TIME CMD
skenny 14815 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 14816 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 15503 15473 7 15:55 pts/27 00:00:08
/soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
/soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
-Dant.home=/soft/apache-ant-1.7.1-r1 -Dant.
skenny 15890 14815 0 15:57 pts/27 00:00:00 ps -f
skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
./nightly.sh: line 588: 14819 Killed "$@" > $OUTPUT 2>&1
+++ ps -f
+++ grep '.*java'
+++ grep -v grep
++ kill_this skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
-DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
-Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
-Djava.security.egd=file:///dev/urandom -classpath
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
skenny 15503 15473 7 15:55 pts/27 00:00:08
/soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
/soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
-Dant.home=/soft/apache-ant-1.7.1-r1
-Dant.library.dir=/soft/apache-ant-1.7.1-r1/lib
org.apache.tools.ant.launch.Launcher -cp :./ -quiet dist
++ '[' -n 14879 ']'
++ /bin/kill -KILL 14879
++ set +x
Executing Package (part 3)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
Executing Package (part 4)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
Executing Package (part 5)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
Executing Package (part 6)
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/swift
Part 1: SSH with PBS and Coasters Configuration Test
Using:
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/sites.template.xml
Using:
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/tc.template.data
`/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/etc/swift.properties'
-> `./swift.properties'
`/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift'
-> `./001-catsn-ssh-pbs-coasters.swift'
Executing
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift
(part 1)
16623 pts/27 00:00:00 nightly.sh
monitor(1): killing test process...
monitor(1): killed process_exec (TERM)
process_exec_trap()
killing all swifts...
++ echo 15473
15473
++ ps -f
UID PID PPID C STIME TTY TIME CMD
skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 16623 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 16624 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
-s groups/group-ssh.sh
skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
skenny 17414 16623 0 16:06 pts/27 00:00:00 ps -f
skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
./nightly.sh: line 588: 16627 Killed "$@" > $OUTPUT 2>&1
+++ ps -f
+++ grep '.*java'
+++ grep -v grep
++ kill_this skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
-DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
-Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
-Djava.security.egd=file:///dev/urandom -classpath
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
++ '[' -n 16687 ']'
++ /bin/kill -KILL 16687
++ set +x
kill 16624: No such process
TOOK: 500
FAILED
Swift svn swift-r3921 (swift modified locally) cog-r3013
RunID: 20110110-1558-ojtlnxfb
Progress:
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
Progress: Selecting site:9 Initializing site shared directory:1
nightly.sh: monitor(1): killed: exceeded 500 seconds
FAILED
++ seq --format %04.f 1 1 10
+ for count in '`seq --format "%04.f" 1 1 10`'
+ '[' -f catsn.0001.out ']'
+ exit 1
----------------------------------------------------------------
i'm running this directly on the pads login and seeing this in the swift
log:
2011-01-10 16:33:18,539-0600 INFO TransportProtocolCommon The Transport
Protocol thread
failed
java.io.IOException: The socket is
EOF
at
com.sshtools.j2ssh.transport.TransportProtocolInputStream.readBufferedData(TransportProtocolInputStream.java:183)
at
com.sshtools.j2ssh.transport.TransportProtocolInputStream.readMessage(TransportProtocolInputStream.java:226)
at
com.sshtools.j2ssh.transport.TransportProtocolCommon.processMessages(TransportProtocolCommon.java:1440)
at
com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1034)
at
com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393)
at
java.lang.Thread.run(Thread.java:619)
you can view the test output here:
http://www.ci.uchicago.edu/~skenny/swift_tests/run-2011-01-10/tests-2011-01-10.html
anyway, thought i'd post this in case there's something that might jump out
at any of you that i can tweak...
~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110110/543e9ffa/attachment.html>
More information about the Swift-devel
mailing list