[Swift-devel] Problems running coaster
Michael Wilde
wilde at mcs.anl.gov
Tue Jul 29 09:38:58 CDT 2008
What are some other possibilities of why the logging code didnt work?
I see the logger.debug calls in the .class file. The logger calls were
mostly unconditional. Possibly a different code path, but less likely.
I will try clearing the cache and re-running.
- Mike
On 7/29/08 9:38 AM, Mihael Hategan wrote:
> There is no order issue. When the service is started the exact list of
> jars to be used is supplied rather than "all jars in this directory".
>
> On Tue, 2008-07-29 at 09:29 -0500, Michael Wilde wrote:
>> I was looking into why my logger.debug statements did not print.
>> I am not sure, but suspect, that the updated jar, loaded into
>> ~/.globus/coasters/cache, was either not placed in the classpath at
>> runtime was was placed after the older copy in the same directory.
>>
>> I have not yet found the logic by which newer classes get loaded to the
>> server, but suspect there may be an issue here. (Or, as usual, pilot
>> error on my part).
>>
>> The class with the updated logging was WorkerManager:
>>
>> [wilde at honest3 cache]$ jar tvf
>> cog-provider-coaster-0.1-a82e2ac11a74fedfadb9a8168a08b6d5.jar | grep
>> WorkerManager
>> 869 Mon Jul 28 19:10:34 CDT 2008
>> org/globus/cog/abstraction/coaster/service/job/manager/WorkerManager$AllocationRequest.class
>> 15556 Mon Jul 28 19:10:34 CDT 2008
>> org/globus/cog/abstraction/coaster/service/job/manager/WorkerManager.class
>> [wilde at honest3 cache]$ jar tvf
>> cog-provider-coaster-0.1-d903eecc754a2c97fb5ceaebdce6ccad.jar | grep
>> WorkerManager
>> 869 Mon Jul 28 23:54:24 CDT 2008
>> org/globus/cog/abstraction/coaster/service/job/manager/WorkerManager$AllocationRequest.class
>> 15963 Mon Jul 28 23:54:24 CDT 2008
>> org/globus/cog/abstraction/coaster/service/job/manager/WorkerManager.class
>> [wilde at honest3 cache]$
>>
>> The *ad.jar file has the correct updated class; the *d5.jar file has the
>> original unmodified class.
>>
>> --
>>
>> If my suspicion about the classpath order is correct, then there is
>> greater possibility that there may be a race in the job launching code
>> of WorkerManager, as this means that the same code hung once and worked
>> once (I'll test more on abe to investigate).
>>
>> - Mike
>>
>>
>>
>> On 7/29/08 12:06 AM, Michael Wilde wrote:
>>> hmmm. my debug statement didnt print. but this time the job on abe ran ok.
>>>
>>> Tomorrow I'll run more tests and see how stable it is there, and why my
>>> logging calls never showed up.
>>>
>>> - Mike
>>>
>>>
>>> On 7/28/08 11:45 PM, Michael Wilde wrote:
>>>> Ive moved on, and put a temp hack in to not use -l and instead run
>>>> "~/.myetcprofile" if it exists and /etc/profile if it doesnt.
>>>>
>>>> .myetcprofile on abe is /etc/profile with the problematic code removed.
>>>>
>>>> Now abe gets past the problem and runs bootstrap.sh ok.
>>>>
>>>> The sequence runs OK up to the point where the service on abe's
>>>> headnode receives a message to start a job.
>>>>
>>>> AT this point, the service on abe seems to hang.
>>>>
>>>> Comparing to the message sequence on mercury, which works, I see this:
>>>>
>>>> *** mercury:
>>>>
>>>> [RequestHandler] DEBUG Channel multiplexer - GSSC-null: HND< 2
>>>> SUBMITJOB(identity=1217268111318
>>>> executable=/bin/bash
>>>> directory=/home/ncsa/wilde/swiftwork/ctest-20080728-1301-7c4ok42h
>>>> arg=shared/wrapper.sh
>>>> arg=echo-myx2e6xi
>>>> arg=-jobdir
>>>> arg=m
>>>> arg=-e
>>>> arg=/bin/echo
>>>> arg=-out
>>>> arg=echo_s000.txt
>>>> arg=-err
>>>> arg=stderr.txt
>>>> arg=-i
>>>> arg=-d
>>>> ar)
>>>> [ChannelManager] DEBUG Channel multiplexer -
>>>> Looking up -134779b6:11b6ad597e2:-7fff:3598cb3d:11b6ad597b5:-7fffS
>>>> [ChannelManager] DEBUG Channel multiplexer - Found
>>>> -134779b6:11b6ad597e2:-7fff:3598cb3d:11b6ad597b5:-7fffS
>>>> [RequestHandler] DEBUG Channel multiplexer - GSSC-null: HND> 2
>>>> SUBMITJOB(urn:1217268111318-1217268128309-1217268128310)
>>>> [Replier] DEBUG Worker 1 - Replier(GSSC-null)REPL>: tag = 2, fin =
>>>> true, datalen = 45, data = urn:1217268111318-1217268128309-1217268128310
>>>> [WorkerManager] INFO Coaster Queue Processor - No suitable worker
>>>> found. Attempting to start a new one.
>>>> [WorkerManager] INFO Worker Manager - Got allocation request:
>>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager$AllocationRequest at 151ca803
>>>>
>>>> [WorkerManager] INFO Worker Manager - Starting worker with
>>>> id=-615912369 and maxwalltime=6060s
>>>> Worker start provider: gt2
>>>> Worker start JM: pbs
>>>>
>>>> *** abe:
>>>>
>>>> [RequestHandler] DEBUG Channel multiplexer - GSSC-null: HND< 2
>>>> SUBMITJOB(identity=1217291444315
>>>> executable=/bin/bash
>>>> directory=/u/ac/wilde/swiftwork/ctest-20080728-1930-m5a70lvc
>>>> arg=shared/wrapper.sh
>>>> arg=echo-zc5mt6xi
>>>> arg=-jobdir
>>>> arg=z
>>>> arg=-e
>>>> arg=/bin/echo
>>>> arg=-out
>>>> arg=echo_s000.txt
>>>> arg=-err
>>>> arg=stderr.txt
>>>> arg=-i
>>>> arg=-d
>>>> arg=
>>>> ar)
>>>> [ChannelManager] DEBUG Channel multiplexer -
>>>> Looking up 17badc64:11b6c39944a:-7fff:f7c31d:11b6c399416:-7fffS
>>>> [ChannelManager] DEBUG Channel multiplexer - Found
>>>> 17badc64:11b6c39944a:-7fff:f7c31d:11b6c399416:-7fffS
>>>> [RequestHandler] DEBUG Channel multiplexer - GSSC-null: HND> 2
>>>> SUBMITJOB(urn:1217291444315-1217291458042-1217291458043)
>>>> [Replier] DEBUG Worker 1 - Replier(GSSC-null)REPL>: tag = 2, fin =
>>>> true, datalen = 45, data = urn:1217291444315-1217291458042-1217291458043
>>>> [WorkerManager] INFO Coaster Queue Processor - No suitable worker
>>>> found. Attempting to start a new one.
>>>> [WorkerManager] INFO Worker Manager - Got allocation request:
>>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager$AllocationRequest at 95cfbe
>>>>
>>>> [AbstractKarajanChannel] DEBUG Channel multiplexer - GSSC-null REQ<:
>>>> tag = 3, fin = true, err = false, datalen = 15, data = SHUTDOWNSERVICE
>>>>
>>>> ***
>>>>
>>>> I *think* the SHUTDOWNSERVICE message on abe is coming much later,
>>>> after abe's service hangs, but Im not sure.
>>>>
>>>> What it looks like to me is that what should should happen on abe is
>>>> this:
>>>>
>>>> [WorkerManager] INFO Worker Manager - Got allocation request:
>>>> org.globus.cog.abstraction.coaster.service.job.manager.WorkerManager$AllocationRequest at 151ca803
>>>>
>>>> [WorkerManager] INFO Worker Manager - Starting worker with
>>>> id=-615912369 and maxwalltime=6060s
>>>>
>>>> but on abe the "Worker Manager - Starting worker" is never seen.
>>>>
>>>> Looking at WorkerManager.run() its hard to see how the "Starting
>>>> worker" message could *not* show up right after "Got allocation
>>>> request", but there must be some sequence of events that causes this.
>>>>
>>>> Abe is an 8-core system. Is there perhaps more opportunity for a
>>>> multi-thread race or deadlock that could cause this?
>>>>
>>>> I will insert some more debug logging and try a few more times to see
>>>> if thing shang in this manner every time or not.
>>>>
>>>> - Mike
>>>>
>>>> ps client Logs with abe server side boot logs are on CI net in
>>>> ~wilde/coast/run11
>>>>
>>>>
>>>>
>>>> On 7/28/08 10:50 PM, Mihael Hategan wrote:
>>>>> On Mon, 2008-07-28 at 19:32 +0000, Ben Clifford wrote:
>>>>>> On Mon, 28 Jul 2008, Michael Wilde wrote:
>>>>>>
>>>>>>> So it looks like something in the job specs that is launching
>>>>>>> coaster for
>>>>>>> gt2:pbs is not being accepted by abe.
>>>>>> ok. TeraGrid's unified account system is insufficiently unified for
>>>>>> me to be able to access abe, but they are aware of that; if and when
>>>>>> I am reunified, I'll try this out myself.
>>>>> Not to be cynical or anything, but that unified thing: never worked.
>>>>>
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
More information about the Swift-devel
mailing list