[Swift-devel] Scripts for persistent and passive coaster configurations

[Michael Wilde]

Hi All,

I'd like Ketan to learn about this aspect of Coasters and to help automate and document the techniques in the process.

Justin, David, can you point out to this list the latest tools, REDMEs, etc you have on the topic?

Ketan, my scripts for doing this are on the CI net under:
  /home/wilde/swift/lab/pecos (for "persistent coasters")

I dont recall what state those scripts were, but I will help you figure them out.

Some background:

Persistent coaster pools ("sites") are useful for:

- running on ad-hoc collections of hosts that you can ssh to
- running on OSG under a "pilot factory job" mechanism
- running on clouds
- running on machines where Globus is not available and the local scheduler is not supported by or does not work well under Swift (eg: Eureka with a deficient Cobalt)
- running on large machines like BG/P or Cray where you want to run many tests within a longer-running fixed partition

"Passive" coaster mode means that the coaster service does not start workers, but rather listens on a port for workers to connect to it.

"Persistent" coasters means that the service, running outside of the Swift JVM (as a separate JVM) is started manually and has a lifetime that persists independent of any Swift clients that connect to it.

Thus the coaster service has two relevant TCP ports:
- a service port by which Swift reaches it (specified in the pool entry)
- a worker port to which workers connect in passive mode

At the moment, the coaster-service command lacks a "-passive" flag to put it in passive mode. Instead, the config script that launches the service needs to run a Swift script using a pool that specifies passive. Once this is done, the coaster service emits its passive port # on stdout/err.

- Mike


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory