[Swift-devel] Failed to start channel GSSCChannel (trunk, coasters, ssh-cl:pbs)

[Mihael Hategan]

Ok, so maybe the ssh-cl provider doesn't properly forward environment
variables. I'll double check that.

On Fri, 2012-02-03 at 14:02 -0600, Thomas Uram wrote:
> I have done this without success:
> 
> GLOBUS_HOSTNAME=fl.ci.uchicago.edu
> GLOBUS_TCP_PORT_RANGE=50000,50100
> swiftt -sites.file sites.coasters.xml -tc.file tc.data hostname.swift
> Swift trunk swift-r5501 (swift modified locally) cog-r3350 (cog modified locally)
> 
> RunID: 20120203-1357-8tekc3f7
> Progress:  time: Fri, 03 Feb 2012 13:57:24 -0600
> Progress:  time: Fri, 03 Feb 2012 13:57:31 -0600  Selecting site:4  Initializing site shared directory:1  Stage in:1
> ssh not set, setting to 'gsissh'
> ssh=gsissh
> Find: https://206.12.24.2:38675
> Find:  keepalive(120), reconnect - https://206.12.24.2:38675
> Progress:  time: Fri, 03 Feb 2012 13:57:35 -0600  Selecting site:4  Submitting:1  Submitted:1
> Failed to transfer wrapper log for job hostname-1jnudkmk
> Progress:  time: Fri, 03 Feb 2012 13:57:38 -0600  Selecting site:3  Stage in:1 Failed but can retry:2
> Failed to transfer wrapper log for job hostname-2jnudkmk
> Failed to transfer wrapper log for job hostname-4jnudkmk
> Progress:  time: Fri, 03 Feb 2012 13:57:54 -0600  Selecting site:3 Failed but can retry:3
> Progress:  time: Fri, 03 Feb 2012 13:57:57 -0600  Selecting site:2  Stage in:1 Failed but can retry:3
> Failed to transfer wrapper log for job hostname-7jnudkmk
> No events in 10s.
> 
> Registered futures:
> ----
> 
> Waiting threads:
> ----
> 
> No events in 10s.
> 
> Registered futures:
> ----
> 
> Waiting threads:
> ----
> 
> ** Ctrl-C here **
> 
> Progress:  time: Fri, 03 Feb 2012 13:58:24 -0600  Selecting site:2 Failed but can retry:4
> Failed to shut down service https://206.12.24.2:38675
> org.globus.cog.karajan.workflow.service.channels.ChannelException: Failed to start channel GSSCChannel-https://206.12.24.2:38675(6)[69518356: {}]
> 	at org.globus.cog.karajan.workflow.service.channels.GSSChannel.reconnect(GSSChannel.java:103)
> 	at org.globus.cog.karajan.workflow.service.channels.GSSChannel.start(GSSChannel.java:62)
> 	at org.globus.cog.karajan.workflow.service.ChannelFactory.newChannel(ChannelFactory.java:55)
> 	at org.globus.cog.karajan.workflow.service.Client.connect(Client.java:116)
> 	at org.globus.cog.karajan.workflow.service.Client.newClient(Client.java:72)
> 	at org.globus.cog.karajan.workflow.service.channels.ChannelManager.connect(ChannelManager.java:236)
> 	at org.globus.cog.karajan.workflow.service.channels.ChannelManager.reserveChannel(ChannelManager.java:256)
> 	at org.globus.cog.karajan.workflow.service.channels.ChannelManager.reserveChannel(ChannelManager.java:217)
> 	at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$ServiceReaper.run(ServiceManager.java:430)
> Caused by: java.net.NoRouteToHostException: No route to host
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> 	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> 	at java.net.Socket.connect(Socket.java:529)
> 	at java.net.Socket.connect(Socket.java:478)
> 	at java.net.Socket.<init>(Socket.java:375)
> 	at java.net.Socket.<init>(Socket.java:276)
> 	at org.globus.net.SocketFactory.createSocket(SocketFactory.java:74)
> 	at org.globus.net.SocketFactory.createSocket(SocketFactory.java:53)
> 	at org.globus.gsi.gssapi.net.GssSocket.<init>(GssSocket.java:56)
> 	at org.globus.gsi.gssapi.net.impl.GSIGssSocket.<init>(GSIGssSocket.java:29)
> 	at org.globus.gsi.gssapi.net.impl.GSIGssSocketFactory.createSocket(GSIGssSocketFactory.java:38)
> 	at org.globus.cog.karajan.workflow.service.channels.GSSChannel.reconnect(GSSChannel.java:89)
> 	... 8 more
> 
> 
> Full log here:
> http://www.mcs.anl.gov/~turam/20120203-1401/hostname-20120203-1357-8tekc3f7.log
> 
> 
> 
> 
> 
> 
> On Feb 3, 2012, at 1:54 PM, Mihael Hategan wrote:
> 
> > On Fri, 2012-02-03 at 13:44 -0600, Thomas Uram wrote:
> >> No I didn't set GLOBUS_HOSTNAME. The address it complains about
> >> (206.12.24.2) is publicly reachable. So is the hostname of the machine
> >> on which I'm running Swift (fl.ci.uchicago.edu).
> > 
> > They should be the same! (i.e. the coaster service tries to connect back
> > to the machine you're running Swift on).
> > 
> > Can you try setting GLOBUS_HOSTNAME and see what happens?
> > 
> >> 
> >> 
> >> I was wondering about the jumble that follows the hostname:port in
> >> that URL:
> >> 
> >> 
> >>>> Failed to start channel
> >>>> GSSCChannel-https://206.12.24.2:35836(2)[1544213635: {}]
> > 
> > (2) is the channel ID
> > [15...] is the channel context
> > They are not part of the IP address, but part of GSSChannel.toString().
> > 
> > 
>