[Swift-devel] Notes from trying new config

[Mihael Hategan]

On Thu, 2014-07-24 at 23:12 -0500, Yadu Nand wrote:
> The issues I raised are mostly resolved:
> 
> 1. I don't see warnings about lazy.errors / lazyErrors,
> 2. I think workDirectory errors were alright, and is better off as a
> required option.

I don't think that it being required was ever under discussion. It was
required, but you would get weird errors at run-time if it wasn't
specified, instead of when the sites file was loaded.

> 3. I got the filesystem.type options for docs.
> The docs also now explain how to use -sites to pick the site from the
> config.
> 
> Now that Mihael is on the thread, I'd like to point out that the nesting
> and braces are excessive.
> In case of execution options, I don't see why there has to be a separate
> group under execution. Same is the case with
> jobOptions nested under options.

you can always say "options.maxJobs: 10" if the braces are a problem.

The reason for "options" is that they create a logical grouping for
optional tunable things for a provider or for a job, but the important
thing about them is that they are provider-specific. I wanted to
separate them from things that apply in general. For example, from
quickly looking at the list of options in 0.95, could you tell whether
the "pe" option applies to a specific provider or all swift jobs? What
about "maxSubmitRate"?

We can discuss this I guess. I thought that the main reason these were
not separated in 0.95 was because they were following the flat structure
of the pre-0.95 sites file.

As for jobOptions, it was previously providerAttributes, which accepted
a string of comma-separated key-value pairs. jobOptions is simply
re-using the existing configuration syntax instead of having its own
syntax. Logically, like jobOptions, providerAttributes was a container
for various settings. 0.95 does not address this issue. Again, I'm happy
to discuss alternatives. Though, honestly, I don't see much of a
difference in typing effort between

providerAttributes="ec2WorkerImage=ami-23700813, ec2WorkerType=t1.micro"

and

jobOptions {ec2WorkerImage: ami-23700813, ec2WorkerType: t1.micro}

Mihael

>  Here's the config I'm using as an example :
> 
> site.cloud {
>     execution {
>         type:"coaster"
>         URL: "127.0.0.1"
>         jobManager: "local:ec2-cloud"
>         options {
>             maxJobs: 10
>             tasksPerNode: 4
>             workerLoggingLevel: TRACE
>             workerLoggingDirectory: /tmp
>             jobOptions {
>                 ec2CredentialsFile:
> ${env.HOME}/.ssh/boto-test-credentials.csv
>                 ec2SecurityGroup: swift_security_group1
>                 ec2KeypairName: swift-test-pair
>                 ec2KeypairFile: ${env.HOME}/swift-test-pair.pem
>                 ec2WorkerImage: ami-23700813
>                 ec2WorkerType: t1.micro
>             }
>         }
>     }
> 
>     # Latest trunk won't run without the next two:
>     execution.retries: 2
>     wrapperlog.always.transfer: False
> 
>     initialParallelTasks: 10
>     maxParallelTasks: 20
>     filesystem.type: swift
>     workDirectory: /tmp/swift-trunk-test
>     staging: "local"
>     app.ALL {executable: "*"}
> }
> 
> sites: cloud