[AG-TECH] VenueServer woes

Thu Jun 29 09:17:01 CDT 2006

Here is an extract from the log files where the problem occurred.

06/28/06 10:38:03 245776 Hosting     Server.py:65 ERROR Exception in SOAP server main loop
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/AccessGrid/hosting/SOAPpy/Server.py", line 63, in Run
    self._server.handle_request()
  File "/usr/lib/python2.4/SocketServer.py", line 217, in handle_request
    request, client_address = self.get_request()
  File "/usr/lib/python2.4/site-packages/SOAPpy/GSIServer.py", line 140, in get_request
    sock, addr = ThreadingGSITCPSocketServer.get_request(self)
  File "/usr/lib/python2.4/site-packages/pyGlobus/io.py", line 846, in get_request
    raise(str(self.socket) + str(" is having trouble listening!:" + str(ex.args)))
<pyGlobus.io.GSITCPSocket instance at 0xb716062c> is having trouble listening!:('an I/O operation was cancelled',)
06/28/06 10:38:03 16384 VenueServer     VenueServer.py:646 INFO                                done.
06/28/06 10:38:03 131081 DataStore     DataStore.py:1629 ERROR Exception handling request
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/AccessGrid/DataStore.py", line 1625, in thread_run
    self.handle_request()
  File "/usr/lib/python2.4/SocketServer.py", line 217, in handle_request
    request, client_address = self.get_request()
  File "/usr/lib/python2.4/site-packages/pyGlobus/io.py", line 846, in get_request
    raise(str(self.socket) + str(" is having trouble listening!:" + str(ex.args)))
<pyGlobus.io.GSITCPSocket instance at 0xb6a65e0c> is having trouble listening!:('an I/O operation was cancelled',)
06/28/06 10:38:03 16384 VenueServer     VenueServer.py:648 INFO  Shutdown Complete.
06/28/06 10:38:03 114696 DataStore     DataStore.py:1555 DEBUG Worker 3 gets cmd quit
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:116 DEBUG After main loop!
06/28/06 10:38:03 98311 DataStore     DataStore.py:1568 DEBUG Worker 2 exiting
06/28/06 10:38:03 16384 Hosting     Server.py:97 ERROR server_close() failed
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/AccessGrid/hosting/SOAPpy/Server.py", line 95, in Stop
    self._server.server_close()
  File "/usr/lib/python2.4/site-packages/pyGlobus/io.py", line 1375, in server_close
    self.socket.close()
  File "/usr/lib/python2.4/site-packages/pyGlobus/io.py", line 1348, in close
    IOBase.close(self)
  File "/usr/lib/python2.4/site-packages/pyGlobus/io.py", line 264, in close
    raise ex
IOBaseException: an uninitialied data structure was passed as argument 1 to globus_io_register_close
06/28/06 10:38:03 114696 DataStore     DataStore.py:1568 DEBUG Worker 3 exiting
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:121 DEBUG Stopped Hosting Environment, exiting.
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <_MainThread(MainThread, started)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <_DummyThread(Dummy-7, started daemon)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <Thread(ConnectionMonitor.Run, started)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <Thread(Thread-23, started)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <_DummyThread(Dummy-24, started daemon)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <_DummyThread(Dummy-25, started daemon)>
06/28/06 10:38:03 16384 Toolkit     VenueServer.py:127 DEBUG Thread <_DummyThread(Dummy-26, started daemon)>
06/28/06 16:46:37 16384 ProxyGen     ProxyGen.py:558 DEBUG Using GT24 proxy code
06/28/06 16:46:37 16384 Toolkit     Toolkit.py:96 DEBUG Initializing AG Toolkit version 2.4 (build 0)
06/28/06 16:46:37 16384 Toolkit     Toolkit.py:138 INFO  Logfile Name: VenueServer.log
06/28/06 16:46:45 16384 Toolkit     Toolkit.py:709 INFO  Service init: have profile None
06/28/06 16:46:45 16384 Toolkit     Config.py:214 DEBUG System hostname of icarus.ichec.ie is valid
06/28/06 16:46:45 16384 CertificateManager     CertificateManager.py:269 DEBUG Opened repository /home/ag2user/.AccessGrid/Config/certRepo
06/28/06 16:46:45 16384 Toolkit     Toolkit.py:738 INFO  Initialized cert mgmt.
06/28/06 16:46:45 16384 Toolkit     Toolkit.py:753 INFO  Loaded profile and configured with it.
06/28/06 16:46:45 16384 CertificateManager     CertificateManager.py:759 DEBUG Configuring standard environment
06/28/06 16:46:45 16384 CertificateManager     CertificateManager.py:827 DEBUG Using default identity /O=Access Grid/O=Argonne National Laboratory/OU=Futures Lab Anonymous Authority/CN=Anonymous User 8dfa3c134968ca0afd2fff7b6e462d25
06/28/06 16:46:45 16384 CertificateManager     CertificateManager.py:1083 DEBUG Initializing environment with unencrypted cert /O=Access Grid/O=Argonne National Laboratory/OU=Futures Lab Anonymous Authority/CN=Anonymous User 8dfa3c134968ca0afd2fff7b6e462d25
06/28/06 16:46:45 16384 CertificateManager     CertificateManager.py:1572 DEBUG done, success=1
06/28/06 16:46:45 16384 Toolkit     Toolkit.py:764 INFO  Initialized Globus.

Cheers,

Niall.
-- 
--
| Niall Moran
| Systems Administrator
| Irish Centre for High-End Computing
 \niall dot moran at ichec dot ie
   --------------------------

On Thu, Jun 29, 2006 at 07:53:07AM -0500, Thomas D. Uram wrote:
> Hi Niall:
> 
> I haven't seen this behavior with a 2.4 server.  Is there any indication 
> of the problem
> in the VenueServer.log file (in ~./AccessGrid/Logs)?
> 
> Tom
> 
> 
> On 6/29/06 4:32 AM, Niall Moran wrote:
> >Hi,
> >
> >I have the accessgrid 2.4 toolkit installed on a Gentoo Linux server.
> >This is used as a VenueServer. 
> >
> >I am having a reoccurring problem where one of the VenueServer processes 
> >goes out of control. It takes up 100% cpu and once this happens the
> >VenueServer will no longer respond. These processes then have to be
> >forcibly killed.
> >
> >This behaviour seems to happen when a client exits the VenueServer. I
> >have been trying to track down what is going on but without much
> >success. Has anyone elso observed this behaviour or experienced
> >something similar?
> >
> >Cheers, 
> >
> >Niall.
> >
> >
> >  
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mcs.anl.gov/pipermail/ag-tech/attachments/20060629/855fa2ab/attachment.pgp>