[Swift-devel] Localhost coasters not working on Beagle
Mihael Hategan
hategan at mcs.anl.gov
Sun Jun 8 22:27:04 CDT 2014
Ok, so:
shell1: hategan at login1:~> netcat -l -p 50003
shell2: hategan at login1:~> netstat -lntp
...
tcp 0 0 0.0.0.0:50003 0.0.0.0:*
LISTEN 22806/netcat
...
hategan at login1:~> telnet 127.0.0.1 50003
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection timed out
I don't think this has anything to do with swift or coasters.
Mihael
On Sun, 2014-06-08 at 20:22 -0700, Mihael Hategan wrote:
> That's odd. Have you tried netstat -lntp? telnet?
>
> I'll give it a shot, but this looks rather strange.
>
> Mihael
>
> On Sun, 2014-06-08 at 22:10 -0500, Michael Wilde wrote:
> > login1$ more /home/wilde/.globus/coasters/worker-0608-0710120-000000.log
> > 2014/06/08 22:07:12.296 INFO - 0608-0710120-000000 Logging started: Sun
> > Jun 8 22:07:12 2014
> > 2014/06/08 22:07:12.296 INFO - Running on node
> > login1.beagle.ci.uchicago.edu
> > 2014/06/08 22:07:12.296 DEBUG - uri=http://127.0.0.1:50003
> > 2014/06/08 22:07:12.296 DEBUG - scheme=http
> > 2014/06/08 22:07:12.297 DEBUG - host=127.0.0.1
> > 2014/06/08 22:07:12.297 DEBUG - port=50003
> > 2014/06/08 22:07:12.297 DEBUG - blockid=0608-0710120-000000
> > 2014/06/08 22:07:12.297 INFO - Connect attempt: 0 ...
> > 2014/06/08 22:07:12.297 INFO - Trying 127.0.0.1:50003 ...
> > 2014/06/08 22:07:33.296 INFO - Connection failed: Connection timed out.
> > Trying other addresses
> > 2014/06/08 22:07:33.296 ERROR - Connection failed for all addresses.
> > 2014/06/08 22:07:33.296 ERROR - Retrying in 1 seconds
> > 2014/06/08 22:07:34.297 INFO - Connect attempt: 1 ...
> > 2014/06/08 22:07:34.297 INFO - Trying 127.0.0.1:50003 ...
> > 2014/06/08 22:07:55.295 INFO - Connection failed: Connection timed out.
> > Trying other addresses
> > 2014/06/08 22:07:55.296 ERROR - Connection failed for all addresses.
> > 2014/06/08 22:07:55.296 ERROR - Retrying in 2 seconds
> > 2014/06/08 22:07:57.298 INFO - Connect attempt: 2 ...
> > 2014/06/08 22:07:57.298 INFO - Trying 127.0.0.1:50003 ...
> > 2014/06/08 22:08:18.295 INFO - Connection failed: Connection timed out.
> > Trying other addresses
> > 2014/06/08 22:08:18.295 ERROR - Connection failed for all addresses.
> > 2014/06/08 22:08:18.295 ERROR - Failed to connect: Connection timed out
> > login1$
> >
> >
> > On 6/8/14, 5:33 PM, Mihael Hategan wrote:
> > > Can you enable worker logging and post the worker log?
> > >
> > > Mihael
> > >
> > > On Sun, 2014-06-08 at 16:48 -0500, Michael Wilde wrote:
> > >> Mihael - Im not able to get a simple localhost coasters run working on
> > >> Beagle login1.
> > >>
> > >> All: Is anyone seeing something similar? It looks to me like my coaster
> > >> worker is not able to connect to the Swift coaster service (using
> > >> standard automatic coasters).
> > >>
> > >> Im working in /lustre/beagle/wilde/swift/lab/fastio (where you can find
> > >> logs and configs). Running 0.95RC6.
> > >>
> > >> Im setting GLOBUS_HOSTNAME (to 127.0.0.1) and have tried internalHost as
> > >> well:
> > >>
> > >> login1$ swift -config cf -tc.file apps -sites.file localcoast.xml
> > >> catsn.swift
> > >>
> > >> login1$ cat localcoast.xml
> > >> <?xml version="1.0" encoding="UTF-8"?>
> > >> <config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites">
> > >>
> > >> <pool handle="localhost">
> > >>
> > >> <execution provider="coaster" jobmanager="local:local"/>
> > >>
> > >> <profile namespace="globus" key="internalHostname">127.0.0.1</profile>
> > >> <profile namespace="globus" key="maxwalltime">00:01:00</profile>
> > >> <profile namespace="globus" key="maxtime">3600</profile>
> > >>
> > >> <profile namespace="globus" key="jobsPerNode">1</profile>
> > >> <profile namespace="globus" key="slots">1</profile>
> > >> <profile namespace="globus" key="nodeGranularity">1</profile>
> > >> <profile namespace="globus" key="maxNodes">1</profile>
> > >>
> > >> <profile namespace="karajan" key="jobThrottle">12</profile>
> > >> <profile namespace="karajan" key="initialScore">10000</profile>
> > >>
> > >> <profile namespace="karajan" key="lowOverAllocation">100</profile>
> > >> <profile namespace="karajan" key="highOverAllocation">100</profile>
> > >>
> > >> <filesystem provider="local"/>
> > >> <workdirectory>/tmp/swiftwork</workdirectory>
> > >>
> > >>
> > >> </pool>
> > >>
> > >> I get error 110 connection timeouts:
> > >>
> > >> 2014-06-08 16:37:50,762-0500 DEBUG swift JOB_START jobid=cat-7jiymsrl
> > >> tr=cat arguments=[data.txt] tmpdir=catsn-run013/jobs/7/cat-7jiymsrl
> > >> host=localhost
> > >> 2014-06-08 16:37:50,829-0500 INFO LocalService Started local service:
> > >> 127.0.0.1:50000
> > >> 2014-06-08 16:37:50,837-0500 INFO BootstrapService Socket bound. URL is
> > >> http://127.0.0.1:50001
> > >> 2014-06-08 16:37:50,914-0500 INFO Settings Local contacts:
> > >> [http://127.0.0.2:50003, http://192.5.86.104:50003,
> > >> http://10.128.2.244:50003]
> > >> 2014-06-08 16:37:50,917-0500 INFO CoasterService Started local service:
> > >> http://127.0.0.1:50003
> > >> 2014-06-08 16:37:50,917-0500 INFO CoasterService Reserving channel for
> > >> registration
> > >> 2014-06-08 16:37:50,942-0500 INFO MetaChannel MetaChannel [context:
> > >> cpipe, boundTo: null] binding to cpipe://1
> > >> 2014-06-08 16:37:50,942-0500 INFO MetaChannel MetaChannel [context:
> > >> spipe, boundTo: null] binding to spipe://1
> > >> 2014-06-08 16:37:50,942-0500 INFO CoasterService Sending registration
> > >> 2014-06-08 16:37:50,948-0500 INFO MetaChannel Trying to re-bind current
> > >> channel
> > >> 2014-06-08 16:37:50,949-0500 INFO RequestHandler Handler(tag: 1,
> > >> REGISTER) unregistering (send)
> > >> 2014-06-08 16:37:50,949-0500 INFO CoasterService Registration complete
> > >> 2014-06-08 16:37:50,949-0500 INFO CoasterService Started coaster
> > >> service: http://127.0.0.1:50002
> > >> 2014-06-08 16:37:50,952-0500 WARN Settings original callback URI is
> > >> http://10.128.2.244:50003
> > >> 2014-06-08 16:37:50,952-0500 WARN Settings callback URI has been
> > >> overridden to http://127.0.0.1:50003
> > >> 2014-06-08 16:37:50,953-0500 INFO RequestHandler Handler(tag: 1,
> > >> CONFIGSERVICE) unregistering (send)
> > >> 2014-06-08 16:37:50,969-0500 INFO BlockQueueProcessor Starting...
> > >> id=0608-3704500
> > >> 2014-06-08 16:37:50,969-0500 INFO RequestHandler Handler(tag: 2,
> > >> SUBMITJOB) unregistering (send)
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor
> > >> Settings {
> > >> slots = 1
> > >> jobsPerNode = 1
> > >> workersPerNode = 1
> > >> nodeGranularity = 1
> > >> allocationStepSize = 0.1
> > >> maxNodes = 1
> > >> lowOverallocation = 10.0
> > >> highOverallocation = 1.0
> > >> overallocationDecayFactor = 0.001
> > >> spread = 0.9
> > >> reserve = 60.000s
> > >> maxtime = 3600
> > >> remoteMonitorEnabled = false
> > >> internalHostname = 127.0.0.1
> > >> hookClass = null
> > >> workerManager = block
> > >> workerLoggingLevel = NONE
> > >> workerLoggingDirectory = DEFAULT
> > >> ldLibraryPath = null
> > >> workerCopies = null
> > >> directory = null
> > >> useHashBang = null
> > >> parallelism = 0.01
> > >> coresPerNode = 1
> > >> perfTraceWorker = false
> > >> perfTraceInterval = -1
> > >> attributes = {}
> > >> callbackURIs = [http://127.0.0.1:50003]
> > >> }
> > >>
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Jobs in holding
> > >> queue: 1
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Time estimate for
> > >> holding queue (seconds): 1
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Allocating blocks
> > >> for a total walltime of: 1s
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Considering:
> > >> Job(id:0 60.000s)
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Max
> > >> Walltime (seconds): 60
> > >> 2014-06-08 16:37:51,009-0500 INFO BlockQueueProcessor Time
> > >> estimate (seconds): 600
> > >> 2014-06-08 16:37:51,010-0500 INFO BlockQueueProcessor Total for
> > >> this new Block (est. seconds): 0
> > >> 2014-06-08 16:37:51,013-0500 INFO BlockQueueProcessor index: 0, last:
> > >> 0, holding.size(): 1
> > >> 2014-06-08 16:37:51,014-0500 INFO BlockQueueProcessor Queued: 1 jobs to
> > >> new Block
> > >> 2014-06-08 16:37:51,014-0500 INFO BlockQueueProcessor index: 0, last:
> > >> 0, ii: 1, holding.size(): 1
> > >> 2014-06-08 16:37:51,014-0500 INFO Block Starting block: workers=1,
> > >> walltime=600.000s
> > >> 2014-06-08 16:37:51,016-0500 INFO RemoteLogHandler BLOCK_REQUESTED
> > >> id=0608-3704500-000000, cores=1, coresPerWorker=1, walltime=600
> > >> 2014-06-08 16:37:51,016-0500 INFO RequestHandler Handler(tag: 2, RLOG)
> > >> unregistering (send)
> > >> 2014-06-08 16:37:51,018-0500 INFO BlockTaskSubmitter Queuing block
> > >> Block 0608-3704500-000000 (1x600.000s) for submission
> > >> 2014-06-08 16:37:51,018-0500 INFO BlockQueueProcessor Added 1 jobs to
> > >> new blocks
> > >> 2014-06-08 16:37:51,018-0500 INFO BlockTaskSubmitter Submitting block
> > >> Block 0608-3704500-000000 (1x600.000s)
> > >> 2014-06-08 16:37:51,018-0500 INFO ExecutionTaskHandler provider=local
> > >> 2014-06-08 16:37:51,023-0500 INFO Block Block task status changed:
> > >> Submitting
> > >> 2014-06-08 16:37:51,023-0500 INFO JobSubmissionTaskHandler Submit: in:
> > >> / command: /usr/bin/perl
> > >> /home/wilde/.globus/coasters/cscript2445623341660096310.pl
> > >> http://127.0.0.1:50003 0608-3704500-000000 NOLOGGING
> > >> 2014-06-08 16:37:51,024-0500 INFO Block Block task status changed:
> > >> Submitted
> > >> 2014-06-08 16:37:51,027-0500 INFO Block Block task status changed: Active
> > >> 2014-06-08 16:37:51,027-0500 INFO RemoteLogHandler BLOCK_ACTIVE
> > >> id=0608-3704500-000000
> > >> 2014-06-08 16:37:51,027-0500 INFO RequestHandler Handler(tag: 3, RLOG)
> > >> unregistering (send)
> > >> 2014-06-08 16:37:51,681-0500 INFO RuntimeStats$ProgressTicker Submitted:1
> > >> 2014-06-08 16:37:51,681-0500 INFO RuntimeStats$ProgressTicker HeapMax:
> > >> 954466304, CrtHeap: 253624320, UsedHeap: 28583112
> > >> 2014-06-08 16:38:21,683-0500 INFO RuntimeStats$ProgressTicker Submitted:1
> > >> 2014-06-08 16:38:21,683-0500 INFO RuntimeStats$ProgressTicker HeapMax:
> > >> 954466304, CrtHeap: 253624320, UsedHeap: 29067208
> > >> 2014-06-08 16:38:51,686-0500 INFO RuntimeStats$ProgressTicker Submitted:1
> > >> 2014-06-08 16:38:51,686-0500 INFO RuntimeStats$ProgressTicker HeapMax:
> > >> 954466304, CrtHeap: 253624320, UsedHeap: 29551304
> > >> 2014-06-08 16:38:57,113-0500 INFO Block Block task status changed:
> > >> Failed Job failed with an exit code of 110
> > >> 2014-06-08 16:38:57,115-0500 INFO Block Failed task spec: Job:
> > >> executable: /usr/bin/perl
> > >> arguments:
> > >> /home/wilde/.globus/coasters/cscript2445623341660096310.pl
> > >> http://127.0.0.1:50003 0608-3704500-000000 NOLOGGING
> > >> stdout: null
> > >> stderr: null
> > >> directory: /
> > >> batch: false
> > >> redirected: false
> > >> attributes:
> > >> hostcount=1,count=1,jobspernode=1,corespernode=1,maxwalltime=10
> > >> env: WORKER_LOGGING_LEVEL=NONE
> > >>
> > >> 2014-06-08 16:38:57,115-0500 INFO Block Worker task failed:
> > >> Failed to connect: Connection timed out at
> > >> /home/wilde/.globus/coasters/cscript2445623341660096310.pl line 1101.
> > >>
> > >>
> > >>
> > >
> >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
More information about the Swift-devel
mailing list