[Swift-devel] running nightly.sh on pads

[Justin M Wozniak]

Hopefully that will do it- the default timeout is only 30 seconds.

I recommend running with -a to skip the ant build, and -p to skip 
something else that is in there.

I will also take a look at why you might be getting the error messages you 
are and try to clean some of that up.

 	Justin

On Mon, 10 Jan 2011, David Kelly wrote:

> Maybe try increasing the time in the .timeout file? I usually see something
> similar when the job exceeds the timeout value
> On Jan 10, 2011 6:17 PM, "Sarah Kenny" <skenny at uchicago.edu> wrote:
>> so, i'm trying to get nightly.sh to run on pads with coasters and i'm not
>> quite sure where this is falling apart. so far the only thing i've edited
> is
>> providers/ssh-pbs-coasters/sites.template.xml (allowing it to take the
>> PROJECT and QUEUE variables). from what i can tell the sites.xml file does
>> get generated correctly but then according to the test output it times out
>> during submission:
>>
>> [skenny at login1 tests]$ ./nightly.sh -c -g -s groups/group-ssh.sh
>> RUNNING_IN:
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10
>> HTML_OUTPUT:
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10/tests-2011-01-10.html
>> which: no ifconfig in
>>
> (/ci/projects/cnari/apps/freesurfer64/bin:/ci/projects/cnari/apps/freesurfer64/fsfast/bin:/ci/projects/cnari/apps/freesurfer64/mni/bin:/ci/projects/cnari/usr/bin:/ci/projects/cnari/apps/afni:/ci/projects/cnari/apps/swift/bin:/soft/java-1.6.0_11-sun-r1/bin:/soft/java-1.6.0_11-sun-r1/jre/bin:/software/common/gx-map-0.5.3.3-r1/bin:/soft/apache-ant-1.7.1-r1/bin:/soft/condor-7.0.5-r1/bin:/soft/globus-4.2.1-r2/bin:/soft/globus-4.2.1-r2/sbin:/usr/kerberos/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/software/common/softenv-1.6.0-r1/bin:/home/skenny/bin/linux-rhel5-x86_64:/home/skenny/bin:/soft/maui-3.2.6p21-r1/bin:/soft/maui-3.2.6p21-r1/sbin:/soft/openmpi-1.4.2-gcc4.1-r1/bin)
>> GROUPLISTFILE: groups/group-ssh.sh
>>
>> Prolog: Build
>>
>> Executing (part 1)
>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
>> Executing (part 2)
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
>> 14815 pts/27 00:00:00 nightly.sh
>> monitor(1): killing test process...
>> touch: cannot touch `killed_test': Stale NFS file handle
>> monitor(1): killed process_exec (TERM)
>> process_exec_trap()
>> killing all swifts...
>> ++ echo 13685
>> 13685
>> ++ ps -f
>> UID PID PPID C STIME TTY TIME CMD
>> skenny 14815 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 14816 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>>
> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
>> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 15503 15473 7 15:55 pts/27 00:00:08
>> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
>> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
>> -Dant.home=/soft/apache-ant-1.7.1-r1 -Dant.
>> skenny 15890 14815 0 15:57 pts/27 00:00:00 ps -f
>> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
>> ./nightly.sh: line 588: 14819 Killed "$@" > $OUTPUT 2>&1
>> +++ ps -f
>> +++ grep '.*java'
>> +++ grep -v grep
>> ++ kill_this skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>>
> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
>> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>>
> login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
> -Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>> -Djava.security.egd=file:///dev/urandom -classpath
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest
 /cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provi
 der-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
 dules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>> skenny 15503 15473 7 15:55 pts/27 00:00:08
>> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
>> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
>> -Dant.home=/soft/apache-ant-1.7.1-r1
>> -Dant.library.dir=/soft/apache-ant-1.7.1-r1/lib
>> org.apache.tools.ant.launch.Launcher -cp :./ -quiet dist
>> ++ '[' -n 14879 ']'
>> ++ /bin/kill -KILL 14879
>> ++ set +x
>> Executing Package (part 3)
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
>> Executing Package (part 4)
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
>> Executing Package (part 5)
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
>> Executing Package (part 6)
>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/swift
>>
>> Part 1: SSH with PBS and Coasters Configuration Test
>>
>> Using:
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/sites.template.xml
>> Using:
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/tc.template.data
>>
> `/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/etc/swift.properties'
>> -> `./swift.properties'
>>
> `/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift'
>> -> `./001-catsn-ssh-pbs-coasters.swift'
>> Executing
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift
>> (part 1)
>> 16623 pts/27 00:00:00 nightly.sh
>> monitor(1): killing test process...
>> monitor(1): killed process_exec (TERM)
>> process_exec_trap()
>> killing all swifts...
>> ++ echo 15473
>> 15473
>> ++ ps -f
>> UID PID PPID C STIME TTY TIME CMD
>> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 16623 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 16624 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>> -s groups/group-ssh.sh
>> skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>>
> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
>> skenny 17414 16623 0 16:06 pts/27 00:00:00 ps -f
>> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
>> ./nightly.sh: line 588: 16627 Killed "$@" > $OUTPUT 2>&1
>> +++ ps -f
>> +++ grep '.*java'
>> +++ grep -v grep
>> ++ kill_this skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>>
> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
>> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>>
> login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
> -Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>> -Djava.security.egd=file:///dev/urandom -classpath
>>
> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest
 /cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provi
 der-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
 dules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>> ++ '[' -n 16687 ']'
>> ++ /bin/kill -KILL 16687
>> ++ set +x
>> kill 16624: No such process
>> TOOK: 500
>> FAILED
>> Swift svn swift-r3921 (swift modified locally) cog-r3013
>>
>> RunID: 20110110-1558-ojtlnxfb
>> Progress:
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> Progress: Selecting site:9 Initializing site shared directory:1
>> nightly.sh: monitor(1): killed: exceeded 500 seconds
>> FAILED
>> ++ seq --format %04.f 1 1 10
>> + for count in '`seq --format "%04.f" 1 1 10`'
>> + '[' -f catsn.0001.out ']'
>> + exit 1
>>
>> ----------------------------------------------------------------
>>
>> i'm running this directly on the pads login and seeing this in the swift
>> log:
>>
>> 2011-01-10 16:33:18,539-0600 INFO TransportProtocolCommon The Transport
>> Protocol thread
>> failed
>>
>> java.io.IOException: The socket is
>> EOF
>>
>> at
>>
> com.sshtools.j2ssh.transport.TransportProtocolInputStream.readBufferedData(TransportProtocolInputStream.java:183)
>>
>> at
>>
> com.sshtools.j2ssh.transport.TransportProtocolInputStream.readMessage(TransportProtocolInputStream.java:226)
>>
>> at
>>
> com.sshtools.j2ssh.transport.TransportProtocolCommon.processMessages(TransportProtocolCommon.java:1440)
>>
>> at
>>
> com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1034)
>>
>> at
>>
> com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393)
>>
>> at
>> java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> you can view the test output here:
>>
>>
> http://www.ci.uchicago.edu/~skenny/swift_tests/run-2011-01-10/tests-2011-01-10.html
>>
>> anyway, thought i'd post this in case there's something that might jump
> out
>> at any of you that i can tweak...
>>
>> ~sk
>

-- 
Justin M Wozniak