[Swift-devel] running nightly.sh on pads

David Kelly dk0966 at cs.ship.edu
Mon Jan 10 18:27:18 CST 2011 

Maybe try increasing the time in the .timeout file? I usually see something
similar when the job exceeds the timeout value
On Jan 10, 2011 6:17 PM, "Sarah Kenny" <skenny at uchicago.edu> wrote:
> so, i'm trying to get nightly.sh to run on pads with coasters and i'm not
> quite sure where this is falling apart. so far the only thing i've edited
is
> providers/ssh-pbs-coasters/sites.template.xml (allowing it to take the
> PROJECT and QUEUE variables). from what i can tell the sites.xml file does
> get generated correctly but then according to the test output it times out
> during submission:
>
> [skenny at login1 tests]$ ./nightly.sh -c -g -s groups/group-ssh.sh
> RUNNING_IN:
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10
> HTML_OUTPUT:
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10/tests-2011-01-10.html
> which: no ifconfig in
>
(/ci/projects/cnari/apps/freesurfer64/bin:/ci/projects/cnari/apps/freesurfer64/fsfast/bin:/ci/projects/cnari/apps/freesurfer64/mni/bin:/ci/projects/cnari/usr/bin:/ci/projects/cnari/apps/afni:/ci/projects/cnari/apps/swift/bin:/soft/java-1.6.0_11-sun-r1/bin:/soft/java-1.6.0_11-sun-r1/jre/bin:/software/common/gx-map-0.5.3.3-r1/bin:/soft/apache-ant-1.7.1-r1/bin:/soft/condor-7.0.5-r1/bin:/soft/globus-4.2.1-r2/bin:/soft/globus-4.2.1-r2/sbin:/usr/kerberos/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/software/common/softenv-1.6.0-r1/bin:/home/skenny/bin/linux-rhel5-x86_64:/home/skenny/bin:/soft/maui-3.2.6p21-r1/bin:/soft/maui-3.2.6p21-r1/sbin:/soft/openmpi-1.4.2-gcc4.1-r1/bin)
> GROUPLISTFILE: groups/group-ssh.sh
>
> Prolog: Build
>
> Executing (part 1)
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
> Executing (part 2)
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
> 14815 pts/27 00:00:00 nightly.sh
> monitor(1): killing test process...
> touch: cannot touch `killed_test': Stale NFS file handle
> monitor(1): killed process_exec (TERM)
> process_exec_trap()
> killing all swifts...
> ++ echo 13685
> 13685
> ++ ps -f
> UID PID PPID C STIME TTY TIME CMD
> skenny 14815 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 14816 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 15503 15473 7 15:55 pts/27 00:00:08
> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
> -Dant.home=/soft/apache-ant-1.7.1-r1 -Dant.
> skenny 15890 14815 0 15:57 pts/27 00:00:00 ps -f
> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
> ./nightly.sh: line 588: 14819 Killed "$@" > $OUTPUT 2>&1
> +++ ps -f
> +++ grep '.*java'
> +++ grep -v grep
> ++ kill_this skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>
login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>
-Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
> -Djava.security.egd=file:///dev/urandom -classpath
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
> skenny 15503 15473 7 15:55 pts/27 00:00:08
> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
> -Dant.home=/soft/apache-ant-1.7.1-r1
> -Dant.library.dir=/soft/apache-ant-1.7.1-r1/lib
> org.apache.tools.ant.launch.Launcher -cp :./ -quiet dist
> ++ '[' -n 14879 ']'
> ++ /bin/kill -KILL 14879
> ++ set +x
> Executing Package (part 3)
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
> Executing Package (part 4)
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
> Executing Package (part 5)
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
> Executing Package (part 6)
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/swift
>
> Part 1: SSH with PBS and Coasters Configuration Test
>
> Using:
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/sites.template.xml
> Using:
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/tc.template.data
>
`/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/etc/swift.properties'
> -> `./swift.properties'
>
`/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift'
> -> `./001-catsn-ssh-pbs-coasters.swift'
> Executing
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift
> (part 1)
> 16623 pts/27 00:00:00 nightly.sh
> monitor(1): killing test process...
> monitor(1): killed process_exec (TERM)
> process_exec_trap()
> killing all swifts...
> ++ echo 15473
> 15473
> ++ ps -f
> UID PID PPID C STIME TTY TIME CMD
> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 16623 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 16624 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
> -s groups/group-ssh.sh
> skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
> skenny 17414 16623 0 16:06 pts/27 00:00:00 ps -f
> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
> ./nightly.sh: line 588: 16627 Killed "$@" > $OUTPUT 2>&1
> +++ ps -f
> +++ grep '.*java'
> +++ grep -v grep
> ++ kill_this skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>
-Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>
login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>
-Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
> -Djava.security.egd=file:///dev/urandom -classpath
>
/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
> ++ '[' -n 16687 ']'
> ++ /bin/kill -KILL 16687
> ++ set +x
> kill 16624: No such process
> TOOK: 500
> FAILED
> Swift svn swift-r3921 (swift modified locally) cog-r3013
>
> RunID: 20110110-1558-ojtlnxfb
> Progress:
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> Progress: Selecting site:9 Initializing site shared directory:1
> nightly.sh: monitor(1): killed: exceeded 500 seconds
> FAILED
> ++ seq --format %04.f 1 1 10
> + for count in '`seq --format "%04.f" 1 1 10`'
> + '[' -f catsn.0001.out ']'
> + exit 1
>
> ----------------------------------------------------------------
>
> i'm running this directly on the pads login and seeing this in the swift
> log:
>
> 2011-01-10 16:33:18,539-0600 INFO TransportProtocolCommon The Transport
> Protocol thread
> failed
>
> java.io.IOException: The socket is
> EOF
>
> at
>
com.sshtools.j2ssh.transport.TransportProtocolInputStream.readBufferedData(TransportProtocolInputStream.java:183)
>
> at
>
com.sshtools.j2ssh.transport.TransportProtocolInputStream.readMessage(TransportProtocolInputStream.java:226)
>
> at
>
com.sshtools.j2ssh.transport.TransportProtocolCommon.processMessages(TransportProtocolCommon.java:1440)
>
> at
>
com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1034)
>
> at
>
com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393)
>
> at
> java.lang.Thread.run(Thread.java:619)
>
>
>
> you can view the test output here:
>
>
http://www.ci.uchicago.edu/~skenny/swift_tests/run-2011-01-10/tests-2011-01-10.html
>
> anyway, thought i'd post this in case there's something that might jump
out
> at any of you that i can tweak...
>
> ~sk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110110/856f55f8/attachment.html>

More information about the Swift-devel mailing list