[Swift-devel] running nightly.sh on pads

[Sarah Kenny]

so i tried again changing the timeout from 500 to 900 and got a slightly
different error in the log.

all the files are here:

http://www.ci.uchicago.edu/~skenny/swift_tests/run-2011-01-11/tests-2011-01-11.html

thought i would go ahead and post to the list in case we wanted to look at
it during the meeting.

~sk

On Mon, Jan 10, 2011 at 4:59 PM, Justin M Wozniak <wozniak at mcs.anl.gov>wrote:

>
> Hopefully that will do it- the default timeout is only 30 seconds.
>
> I recommend running with -a to skip the ant build, and -p to skip something
> else that is in there.
>
> I will also take a look at why you might be getting the error messages you
> are and try to clean some of that up.
>
>        Justin
>
>
> On Mon, 10 Jan 2011, David Kelly wrote:
>
>  Maybe try increasing the time in the .timeout file? I usually see
>> something
>> similar when the job exceeds the timeout value
>> On Jan 10, 2011 6:17 PM, "Sarah Kenny" <skenny at uchicago.edu> wrote:
>>
>>> so, i'm trying to get nightly.sh to run on pads with coasters and i'm not
>>> quite sure where this is falling apart. so far the only thing i've edited
>>>
>> is
>>
>>> providers/ssh-pbs-coasters/sites.template.xml (allowing it to take the
>>> PROJECT and QUEUE variables). from what i can tell the sites.xml file
>>> does
>>> get generated correctly but then according to the test output it times
>>> out
>>> during submission:
>>>
>>> [skenny at login1 tests]$ ./nightly.sh -c -g -s groups/group-ssh.sh
>>> RUNNING_IN:
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10
>>
>>> HTML_OUTPUT:
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/run-2011-01-10/tests-2011-01-10.html
>>
>>> which: no ifconfig in
>>>
>>> (/ci/projects/cnari/apps/freesurfer64/bin:/ci/projects/cnari/apps/freesurfer64/fsfast/bin:/ci/projects/cnari/apps/freesurfer64/mni/bin:/ci/projects/cnari/usr/bin:/ci/projects/cnari/apps/afni:/ci/projects/cnari/apps/swift/bin:/soft/java-1.6.0_11-sun-r1/bin:/soft/java-1.6.0_11-sun-r1/jre/bin:/software/common/gx-map-0.5.3.3-r1/bin:/soft/apache-ant-1.7.1-r1/bin:/soft/condor-7.0.5-r1/bin:/soft/globus-4.2.1-r2/bin:/soft/globus-4.2.1-r2/sbin:/usr/kerberos/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/software/common/softenv-1.6.0-r1/bin:/home/skenny/bin/linux-rhel5-x86_64:/home/skenny/bin:/soft/maui-3.2.6p21-r1/bin:/soft/maui-3.2.6p21-r1/sbin:/soft/openmpi-1.4.2-gcc4.1-r1/bin)
>>
>>> GROUPLISTFILE: groups/group-ssh.sh
>>>
>>> Prolog: Build
>>>
>>> Executing (part 1)
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
>>> Executing (part 2)
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
>>
>>> 14815 pts/27 00:00:00 nightly.sh
>>> monitor(1): killing test process...
>>> touch: cannot touch `killed_test': Stale NFS file handle
>>> monitor(1): killed process_exec (TERM)
>>> process_exec_trap()
>>> killing all swifts...
>>> ++ echo 13685
>>> 13685
>>> ++ ps -f
>>> UID PID PPID C STIME TTY TIME CMD
>>> skenny 14815 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 14816 1 0 15:49 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>>>
>>> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
>>
>>> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 15503 15473 7 15:55 pts/27 00:00:08
>>> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
>>> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
>>> -Dant.home=/soft/apache-ant-1.7.1-r1 -Dant.
>>> skenny 15890 14815 0 15:57 pts/27 00:00:00 ps -f
>>> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
>>> ./nightly.sh: line 588: 14819 Killed "$@" > $OUTPUT 2>&1
>>> +++ ps -f
>>> +++ grep '.*java'
>>> +++ grep -v grep
>>> ++ kill_this skenny 14879 1 0 15:49 pts/27 00:00:04 java -Xmx2048M
>>>
>>> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
>>
>>> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>>>
>>> login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
>>>
>>> -Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
>>> -Djava.security.egd=file:///dev/urandom -classpath
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest
>>
>
> /cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provi
>
> der-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>
> dules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>
>> skenny 15503 15473 7 15:55 pts/27 00:00:08
>>> /soft/java-1.6.0_11-sun-r1/jre/bin/java -classpath
>>> /soft/apache-ant-1.7.1-r1/lib/ant-launcher.jar
>>> -Dant.home=/soft/apache-ant-1.7.1-r1
>>> -Dant.library.dir=/soft/apache-ant-1.7.1-r1/lib
>>> org.apache.tools.ant.launch.Launcher -cp :./ -quiet dist
>>> ++ '[' -n 14879 ']'
>>> ++ /bin/kill -KILL 14879
>>> ++ set +x
>>> Executing Package (part 3)
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift
>>
>>> Executing Package (part 4)
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
>>
>>> Executing Package (part 5)
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/lib
>>
>>> Executing Package (part 6)
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/swift
>>
>>>
>>> Part 1: SSH with PBS and Coasters Configuration Test
>>>
>>> Using:
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/sites.template.xml
>>
>>> Using:
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/tc.template.data
>>
>>>
>>> `/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/etc/swift.properties'
>>
>>> -> `./swift.properties'
>>>
>>> `/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift'
>>
>>> -> `./001-catsn-ssh-pbs-coasters.swift'
>>> Executing
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/tests/providers/ssh-pbs-coasters/001-catsn-ssh-pbs-coasters.swift
>>
>>> (part 1)
>>> 16623 pts/27 00:00:00 nightly.sh
>>> monitor(1): killing test process...
>>> monitor(1): killed process_exec (TERM)
>>> process_exec_trap()
>>> killing all swifts...
>>> ++ echo 15473
>>> 15473
>>> ++ ps -f
>>> UID PID PPID C STIME TTY TIME CMD
>>> skenny 15473 23767 0 15:55 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 16623 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 16624 15473 0 15:58 pts/27 00:00:00 /bin/bash ./nightly.sh -c -g
>>> -s groups/group-ssh.sh
>>> skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>>>
>>> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/
>>
>>> skenny 17414 16623 0 16:06 pts/27 00:00:00 ps -f
>>> skenny 23767 23760 0 13:53 pts/27 00:00:00 -bash
>>> ./nightly.sh: line 588: 16627 Killed "$@" > $OUTPUT 2>&1
>>> +++ ps -f
>>> +++ grep '.*java'
>>> +++ grep -v grep
>>> ++ kill_this skenny 16687 1 0 15:58 pts/27 00:00:04 java -Xmx2048M
>>>
>>> -Djava.endorsed.dirs=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/endorsed
>>
>>> -DUID=1195 -DGLOBUS_TCP_PORT_RANGE=50000,51000 -DGLOBUS_HOSTNAME=
>>>
>>> login1.pads.ci.uchicago.edu-DCOG_INSTALL_PATH=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
>>>
>>> -Dswift.home=/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/..
>>
>>> -Djava.security.egd=file:///dev/urandom -classpath
>>>
>>> /ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../etc:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../libexec:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/addressing-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/antlr-2.7.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/axis-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/backport-util-concurrent.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/coaster-bootstrap.jar:/ci/projects/cnari/soft/swift_latest
>>
>
> /cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-abstraction-common-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-axis.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-grapheditor-0.47.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-jglobus-1.7.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-karajan-0.36-dev.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-clref-gt4_0_0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-coaster-0.3.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provi
>
> der-dcache-0.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt2-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-gt4_0_0-2.5.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-local-2.2.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-localscheduler-0.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-ssh-2.4.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-provider-webdav-2.1.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-resources-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>
> dules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-swift-svn.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-trap-1.0.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-url.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/cog-util-0.92.jar:/ci/projects/cnari/soft/swift_latest/cog/modules/swift/tests/cog/modules/swift/dist/swift-svn/bin/../lib/commonj.jar:/ci/projects/cnari/soft/swift_latest/cog/mo
>
>> ++ '[' -n 16687 ']'
>>> ++ /bin/kill -KILL 16687
>>> ++ set +x
>>> kill 16624: No such process
>>> TOOK: 500
>>> FAILED
>>> Swift svn swift-r3921 (swift modified locally) cog-r3013
>>>
>>> RunID: 20110110-1558-ojtlnxfb
>>> Progress:
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> Progress: Selecting site:9 Initializing site shared directory:1
>>> nightly.sh: monitor(1): killed: exceeded 500 seconds
>>> FAILED
>>> ++ seq --format %04.f 1 1 10
>>> + for count in '`seq --format "%04.f" 1 1 10`'
>>> + '[' -f catsn.0001.out ']'
>>> + exit 1
>>>
>>> ----------------------------------------------------------------
>>>
>>> i'm running this directly on the pads login and seeing this in the swift
>>> log:
>>>
>>> 2011-01-10 16:33:18,539-0600 INFO TransportProtocolCommon The Transport
>>> Protocol thread
>>> failed
>>>
>>> java.io.IOException: The socket is
>>> EOF
>>>
>>> at
>>>
>>> com.sshtools.j2ssh.transport.TransportProtocolInputStream.readBufferedData(TransportProtocolInputStream.java:183)
>>
>>>
>>> at
>>>
>>> com.sshtools.j2ssh.transport.TransportProtocolInputStream.readMessage(TransportProtocolInputStream.java:226)
>>
>>>
>>> at
>>>
>>> com.sshtools.j2ssh.transport.TransportProtocolCommon.processMessages(TransportProtocolCommon.java:1440)
>>
>>>
>>> at
>>>
>>> com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1034)
>>
>>>
>>> at
>>>
>>> com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393)
>>
>>>
>>> at
>>> java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>>
>>> you can view the test output here:
>>>
>>>
>>>
>> http://www.ci.uchicago.edu/~skenny/swift_tests/run-2011-01-10/tests-2011-01-10.html<http://www.ci.uchicago.edu/%7Eskenny/swift_tests/run-2011-01-10/tests-2011-01-10.html>
>>
>>>
>>> anyway, thought i'd post this in case there's something that might jump
>>>
>> out
>>
>>> at any of you that i can tweak...
>>>
>>> ~sk
>>>
>>
>>
> --
> Justin M Wozniak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20110111/145878da/attachment.html>