From hategan at mcs.anl.gov  Fri Jul  4 03:07:26 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 4 Jul 2014 01:07:26 -0700
Subject: [Swift-devel] changes
Message-ID: <1404461246.11929.27.camel@echo>

Hi,

I committed a fairly large number of changes. Some things got limited
testing, and some are work in progress. For example, I need to update
sites files used in tests, but it's getting late here.
Anyway, here's a summary:

1. New sites.xml format and removal of tc.data. I expect a bunch of
things to break, but we discussed cleaning both sites.xml and tc.data
many times and I guess I never got the courage to do it until now. This
format is not backwards compatible (intentionally). It goes like this:
<sites>
  <apps>? <!-- global apps -->
    <property name="...">...</property>* <!-- global app properties -->
    <env name="...">...</env>* <!-- global app envs -->
    <app name="..." executable="...">*
       <property/>*
       <env/>*
    </app>
  </apps>

  <site name="...">
    <execution provider="..."...>
      <property/>*
    </execution>
    <filesystem>?
     <property/>*
    </filesystem>
    <wokdirectory/>
    <scratch/>?
    <apps/>?
  <site>
</sites>

There are no more namespaces since they were mostly used to figure out
whether a property was going to the task or the site. I also changed
"profile" to "property", since "profile" seemed a bit arcane.

You should also now be able to use maxParallelTasks and
initialParallelTasks instead of jobThrottle and initialScore. It
computes the latter automatically.

2. Coasters now support multiple configurations. So you can have two
local:local coaster sites with different settings and it should work.
The way this works is that every configuration (and every different run)
gets its own job queue, settings, and block allocator, and they don't
interact with each other. This might be problematic with things like
slots which is meant to limit the number of jobs globally, but that's a
minor annoyance. 

3. Passive workers can be launched with a -c(oncurrency) argument (1 by
default). This will be the jobs per node setting. Each worker can have a
different number, and that should work as expected. For passive workers,
the client side jobsPerNode setting will be ignored.

4. There is a tool to analyze logs now (swift-log-info). This is simply
an offline version of the http monitor. It can parse logs and feed the
information to a web browser. It should also be able to follow logs as
they are produced and feed live information to a browser. It can also
fake a live log by parsing it slowly instead of all-at-once.

5. The http monitor got a bit of an update. You can look at all kinds of
statistics, browse through apps, look at pretty plots of how apps
behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
fashion. It should be cute and useful in one. Won't work with older
logs. Some pics attached.

6. There are now worker.pl "probes" for CPU usage, disk usage, and some
I/O stats. If you click on a link to a worker in the http monitor,
you'll get nice graphs of these things. The probes are currently
hardcoded to run every 60 seconds (although in the pictures below they
ran at 1/s).

Mihael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s1.png
Type: image/png
Size: 65562 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/b97adb11/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: s2.png
Type: image/png
Size: 72654 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/b97adb11/attachment-0001.png>

From wilde at anl.gov  Fri Jul  4 11:02:46 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 4 Jul 2014 11:02:46 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404461246.11929.27.camel@echo>
References: <1404461246.11929.27.camel@echo>
Message-ID: <53B6D026.2030205@anl.gov>

A lot of great work, here, Mihael - very nice!

We'll need to re-adjust the new config mechanism to match this, unless 
you've already done that.

This means that we'll need to get all users converted to the new config 
before this gets released - else they will have no backwards 
compatibility for their current configurations (sites and tc files).

Nonetheless, as we discussed in a prior meeting, I think we agreed that 
this was a necessary step and the right way to go.

- Mike

On 7/4/14, 3:07 AM, Mihael Hategan wrote:
> Hi,
>
> I committed a fairly large number of changes. Some things got limited
> testing, and some are work in progress. For example, I need to update
> sites files used in tests, but it's getting late here.
> Anyway, here's a summary:
>
> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> things to break, but we discussed cleaning both sites.xml and tc.data
> many times and I guess I never got the courage to do it until now. This
> format is not backwards compatible (intentionally). It goes like this:
> <sites>
>    <apps>? <!-- global apps -->
>      <property name="...">...</property>* <!-- global app properties -->
>      <env name="...">...</env>* <!-- global app envs -->
>      <app name="..." executable="...">*
>         <property/>*
>         <env/>*
>      </app>
>    </apps>
>
>    <site name="...">
>      <execution provider="..."...>
>        <property/>*
>      </execution>
>      <filesystem>?
>       <property/>*
>      </filesystem>
>      <wokdirectory/>
>      <scratch/>?
>      <apps/>?
>    <site>
> </sites>
>
> There are no more namespaces since they were mostly used to figure out
> whether a property was going to the task or the site. I also changed
> "profile" to "property", since "profile" seemed a bit arcane.
>
> You should also now be able to use maxParallelTasks and
> initialParallelTasks instead of jobThrottle and initialScore. It
> computes the latter automatically.
>
> 2. Coasters now support multiple configurations. So you can have two
> local:local coaster sites with different settings and it should work.
> The way this works is that every configuration (and every different run)
> gets its own job queue, settings, and block allocator, and they don't
> interact with each other. This might be problematic with things like
> slots which is meant to limit the number of jobs globally, but that's a
> minor annoyance.
>
> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> default). This will be the jobs per node setting. Each worker can have a
> different number, and that should work as expected. For passive workers,
> the client side jobsPerNode setting will be ignored.
>
> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> an offline version of the http monitor. It can parse logs and feed the
> information to a web browser. It should also be able to follow logs as
> they are produced and feed live information to a browser. It can also
> fake a live log by parsing it slowly instead of all-at-once.
>
> 5. The http monitor got a bit of an update. You can look at all kinds of
> statistics, browse through apps, look at pretty plots of how apps
> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> fashion. It should be cute and useful in one. Won't work with older
> logs. Some pics attached.
>
> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> I/O stats. If you click on a link to a worker in the http monitor,
> you'll get nice graphs of these things. The probes are currently
> hardcoded to run every 60 seconds (although in the pictures below they
> ran at 1/s).
>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/d9007f9f/attachment.html>

From tim.g.armstrong at gmail.com  Fri Jul  4 11:49:11 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Fri, 4 Jul 2014 11:49:11 -0500
Subject: [Swift-devel] changes
In-Reply-To: <53B6D026.2030205@anl.gov>
References: <1404461246.11929.27.camel@echo>
	<53B6D026.2030205@anl.gov>
Message-ID: <CAC0jiV68Uh2QxhRRYKvWSv7X5C_TKEt036Mrj2vf9E+Ex472ZQ@mail.gmail.com>

I had a semi-related question: is there any documentation aside from the
user guide about sites, providers, and job managers in Swift/Coasters?

- Tim


On Fri, Jul 4, 2014 at 11:02 AM, Michael Wilde <wilde at anl.gov> wrote:

>  A lot of great work, here, Mihael - very nice!
>
> We'll need to re-adjust the new config mechanism to match this, unless
> you've already done that.
>
> This means that we'll need to get all users converted to the new config
> before this gets released - else they will have no backwards compatibility
> for their current configurations (sites and tc files).
>
> Nonetheless, as we discussed in a prior meeting, I think we agreed that
> this was a necessary step and the right way to go.
>
> - Mike
>
>
> On 7/4/14, 3:07 AM, Mihael Hategan wrote:
>
> Hi,
>
> I committed a fairly large number of changes. Some things got limited
> testing, and some are work in progress. For example, I need to update
> sites files used in tests, but it's getting late here.
> Anyway, here's a summary:
>
> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> things to break, but we discussed cleaning both sites.xml and tc.data
> many times and I guess I never got the courage to do it until now. This
> format is not backwards compatible (intentionally). It goes like this:
> <sites>
>   <apps>? <!-- global apps -->
>     <property name="...">...</property>* <!-- global app properties -->
>     <env name="...">...</env>* <!-- global app envs -->
>     <app name="..." executable="...">*
>        <property/>*
>        <env/>*
>     </app>
>   </apps>
>
>   <site name="...">
>     <execution provider="..."...>
>       <property/>*
>     </execution>
>     <filesystem>?
>      <property/>*
>     </filesystem>
>     <wokdirectory/>
>     <scratch/>?
>     <apps/>?
>   <site>
> </sites>
>
> There are no more namespaces since they were mostly used to figure out
> whether a property was going to the task or the site. I also changed
> "profile" to "property", since "profile" seemed a bit arcane.
>
> You should also now be able to use maxParallelTasks and
> initialParallelTasks instead of jobThrottle and initialScore. It
> computes the latter automatically.
>
> 2. Coasters now support multiple configurations. So you can have two
> local:local coaster sites with different settings and it should work.
> The way this works is that every configuration (and every different run)
> gets its own job queue, settings, and block allocator, and they don't
> interact with each other. This might be problematic with things like
> slots which is meant to limit the number of jobs globally, but that's a
> minor annoyance.
>
> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> default). This will be the jobs per node setting. Each worker can have a
> different number, and that should work as expected. For passive workers,
> the client side jobsPerNode setting will be ignored.
>
> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> an offline version of the http monitor. It can parse logs and feed the
> information to a web browser. It should also be able to follow logs as
> they are produced and feed live information to a browser. It can also
> fake a live log by parsing it slowly instead of all-at-once.
>
> 5. The http monitor got a bit of an update. You can look at all kinds of
> statistics, browse through apps, look at pretty plots of how apps
> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> fashion. It should be cute and useful in one. Won't work with older
> logs. Some pics attached.
>
> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> I/O stats. If you click on a link to a worker in the http monitor,
> you'll get nice graphs of these things. The probes are currently
> hardcoded to run every 60 seconds (although in the pictures below they
> ran at 1/s).
>
> Mihael
>
>
>
> _______________________________________________
> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> --
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/e0e144f9/attachment.html>

From wilde at anl.gov  Fri Jul  4 12:08:12 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 4 Jul 2014 12:08:12 -0500
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV68Uh2QxhRRYKvWSv7X5C_TKEt036Mrj2vf9E+Ex472ZQ@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>	<53B6D026.2030205@anl.gov>
	<CAC0jiV68Uh2QxhRRYKvWSv7X5C_TKEt036Mrj2vf9E+Ex472ZQ@mail.gmail.com>
Message-ID: <53B6DF7C.40602@anl.gov>

Tim, there's the Site Guide which covers this in a per-site manner:

http://swift-lang.org/guides/trunk/siteguide/siteguide.html

We need to revise this material and integrate more of the concepts into 
the User Guide.

- Mike

On 7/4/14, 11:49 AM, Tim Armstrong wrote:
> I had a semi-related question: is there any documentation aside from 
> the user guide about sites, providers, and job managers in Swift/Coasters?
>
> - Tim
>
>
> On Fri, Jul 4, 2014 at 11:02 AM, Michael Wilde <wilde at anl.gov 
> <mailto:wilde at anl.gov>> wrote:
>
>     A lot of great work, here, Mihael - very nice!
>
>     We'll need to re-adjust the new config mechanism to match this,
>     unless you've already done that.
>
>     This means that we'll need to get all users converted to the new
>     config before this gets released - else they will have no
>     backwards compatibility for their current configurations (sites
>     and tc files).
>
>     Nonetheless, as we discussed in a prior meeting, I think we agreed
>     that this was a necessary step and the right way to go.
>
>     - Mike
>
>
>     On 7/4/14, 3:07 AM, Mihael Hategan wrote:
>>     Hi,
>>
>>     I committed a fairly large number of changes. Some things got limited
>>     testing, and some are work in progress. For example, I need to update
>>     sites files used in tests, but it's getting late here.
>>     Anyway, here's a summary:
>>
>>     1. New sites.xml format and removal of tc.data. I expect a bunch of
>>     things to break, but we discussed cleaning both sites.xml and tc.data
>>     many times and I guess I never got the courage to do it until now. This
>>     format is not backwards compatible (intentionally). It goes like this:
>>     <sites>
>>        <apps>? <!-- global apps -->
>>          <property name="...">...</property>* <!-- global app properties -->
>>          <env name="...">...</env>* <!-- global app envs -->
>>          <app name="..." executable="...">*
>>             <property/>*
>>             <env/>*
>>          </app>
>>        </apps>
>>
>>        <site name="...">
>>          <execution provider="..."...>
>>            <property/>*
>>          </execution>
>>          <filesystem>?
>>           <property/>*
>>          </filesystem>
>>          <wokdirectory/>
>>          <scratch/>?
>>          <apps/>?
>>        <site>
>>     </sites>
>>
>>     There are no more namespaces since they were mostly used to figure out
>>     whether a property was going to the task or the site. I also changed
>>     "profile" to "property", since "profile" seemed a bit arcane.
>>
>>     You should also now be able to use maxParallelTasks and
>>     initialParallelTasks instead of jobThrottle and initialScore. It
>>     computes the latter automatically.
>>
>>     2. Coasters now support multiple configurations. So you can have two
>>     local:local coaster sites with different settings and it should work.
>>     The way this works is that every configuration (and every different run)
>>     gets its own job queue, settings, and block allocator, and they don't
>>     interact with each other. This might be problematic with things like
>>     slots which is meant to limit the number of jobs globally, but that's a
>>     minor annoyance.
>>
>>     3. Passive workers can be launched with a -c(oncurrency) argument (1 by
>>     default). This will be the jobs per node setting. Each worker can have a
>>     different number, and that should work as expected. For passive workers,
>>     the client side jobsPerNode setting will be ignored.
>>
>>     4. There is a tool to analyze logs now (swift-log-info). This is simply
>>     an offline version of the http monitor. It can parse logs and feed the
>>     information to a web browser. It should also be able to follow logs as
>>     they are produced and feed live information to a browser. It can also
>>     fake a live log by parsing it slowly instead of all-at-once.
>>
>>     5. The http monitor got a bit of an update. You can look at all kinds of
>>     statistics, browse through apps, look at pretty plots of how apps
>>     behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>>     fashion. It should be cute and useful in one. Won't work with older
>>     logs. Some pics attached.
>>
>>     6. There are nowworker.pl  <http://worker.pl>  "probes" for CPU usage, disk usage, and some
>>     I/O stats. If you click on a link to a worker in the http monitor,
>>     you'll get nice graphs of these things. The probes are currently
>>     hardcoded to run every 60 seconds (although in the pictures below they
>>     ran at 1/s).
>>
>>     Mihael
>>
>>
>>     _______________________________________________
>>     Swift-devel mailing list
>>     Swift-devel at ci.uchicago.edu  <mailto:Swift-devel at ci.uchicago.edu>
>>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>     -- 
>     Michael Wilde
>     Mathematics and Computer Science          Computation Institute
>     Argonne National Laboratory               The University of Chicago
>
>
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/5f7d8029/attachment.html>

From hategan at mcs.anl.gov  Fri Jul  4 12:09:29 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 4 Jul 2014 10:09:29 -0700
Subject: [Swift-devel] changes
In-Reply-To: <53B6D026.2030205@anl.gov>
References: <1404461246.11929.27.camel@echo> <53B6D026.2030205@anl.gov>
Message-ID: <1404493769.20087.6.camel@echo>

On Fri, 2014-07-04 at 11:02 -0500, Michael Wilde wrote:
> A lot of great work, here, Mihael - very nice!

Thanks.

> 
> We'll need to re-adjust the new config mechanism to match this, unless 
> you've already done that.

I have not. I wanted to sync my changes first.

> 
> This means that we'll need to get all users converted to the new config 
> before this gets released - else they will have no backwards 
> compatibility for their current configurations (sites and tc files).

I think you suggested and I very much agreed that a tool to upgrade the
sites file will be there. I have not yet written it, but it's coming
today-ish.

> 
> Nonetheless, as we discussed in a prior meeting, I think we agreed that 
> this was a necessary step and the right way to go.

I have my doubts about "necessary". I know I argued that previously.
Technically speaking, if I can write a conversion tool, then it can be
done automatically under the covers.
But I think it was a step in the right direction.

Mihael



From hategan at mcs.anl.gov  Fri Jul  4 12:10:42 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 4 Jul 2014 10:10:42 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV68Uh2QxhRRYKvWSv7X5C_TKEt036Mrj2vf9E+Ex472ZQ@mail.gmail.com>
References: <1404461246.11929.27.camel@echo> <53B6D026.2030205@anl.gov>
	<CAC0jiV68Uh2QxhRRYKvWSv7X5C_TKEt036Mrj2vf9E+Ex472ZQ@mail.gmail.com>
Message-ID: <1404493842.20087.7.camel@echo>

No, but I think this is a good opportunity for me to write one.

Mihael

On Fri, 2014-07-04 at 11:49 -0500, Tim Armstrong wrote:
> I had a semi-related question: is there any documentation aside from the
> user guide about sites, providers, and job managers in Swift/Coasters?
> 
> - Tim
> 
> 
> On Fri, Jul 4, 2014 at 11:02 AM, Michael Wilde <wilde at anl.gov> wrote:
> 
> >  A lot of great work, here, Mihael - very nice!
> >
> > We'll need to re-adjust the new config mechanism to match this, unless
> > you've already done that.
> >
> > This means that we'll need to get all users converted to the new config
> > before this gets released - else they will have no backwards compatibility
> > for their current configurations (sites and tc files).
> >
> > Nonetheless, as we discussed in a prior meeting, I think we agreed that
> > this was a necessary step and the right way to go.
> >
> > - Mike
> >
> >
> > On 7/4/14, 3:07 AM, Mihael Hategan wrote:
> >
> > Hi,
> >
> > I committed a fairly large number of changes. Some things got limited
> > testing, and some are work in progress. For example, I need to update
> > sites files used in tests, but it's getting late here.
> > Anyway, here's a summary:
> >
> > 1. New sites.xml format and removal of tc.data. I expect a bunch of
> > things to break, but we discussed cleaning both sites.xml and tc.data
> > many times and I guess I never got the courage to do it until now. This
> > format is not backwards compatible (intentionally). It goes like this:
> > <sites>
> >   <apps>? <!-- global apps -->
> >     <property name="...">...</property>* <!-- global app properties -->
> >     <env name="...">...</env>* <!-- global app envs -->
> >     <app name="..." executable="...">*
> >        <property/>*
> >        <env/>*
> >     </app>
> >   </apps>
> >
> >   <site name="...">
> >     <execution provider="..."...>
> >       <property/>*
> >     </execution>
> >     <filesystem>?
> >      <property/>*
> >     </filesystem>
> >     <wokdirectory/>
> >     <scratch/>?
> >     <apps/>?
> >   <site>
> > </sites>
> >
> > There are no more namespaces since they were mostly used to figure out
> > whether a property was going to the task or the site. I also changed
> > "profile" to "property", since "profile" seemed a bit arcane.
> >
> > You should also now be able to use maxParallelTasks and
> > initialParallelTasks instead of jobThrottle and initialScore. It
> > computes the latter automatically.
> >
> > 2. Coasters now support multiple configurations. So you can have two
> > local:local coaster sites with different settings and it should work.
> > The way this works is that every configuration (and every different run)
> > gets its own job queue, settings, and block allocator, and they don't
> > interact with each other. This might be problematic with things like
> > slots which is meant to limit the number of jobs globally, but that's a
> > minor annoyance.
> >
> > 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> > default). This will be the jobs per node setting. Each worker can have a
> > different number, and that should work as expected. For passive workers,
> > the client side jobsPerNode setting will be ignored.
> >
> > 4. There is a tool to analyze logs now (swift-log-info). This is simply
> > an offline version of the http monitor. It can parse logs and feed the
> > information to a web browser. It should also be able to follow logs as
> > they are produced and feed live information to a browser. It can also
> > fake a live log by parsing it slowly instead of all-at-once.
> >
> > 5. The http monitor got a bit of an update. You can look at all kinds of
> > statistics, browse through apps, look at pretty plots of how apps
> > behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> > fashion. It should be cute and useful in one. Won't work with older
> > logs. Some pics attached.
> >
> > 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> > I/O stats. If you click on a link to a worker in the http monitor,
> > you'll get nice graphs of these things. The probes are currently
> > hardcoded to run every 60 seconds (although in the pictures below they
> > ran at 1/s).
> >
> > Mihael
> >
> >
> >
> > _______________________________________________
> > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > --
> > Michael Wilde
> > Mathematics and Computer Science          Computation Institute
> > Argonne National Laboratory               The University of Chicago
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From davidkelly at uchicago.edu  Fri Jul  4 18:05:06 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Fri, 4 Jul 2014 18:05:06 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404461246.11929.27.camel@echo>
References: <1404461246.11929.27.camel@echo>
Message-ID: <CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>

This looks great - lots of very useful changes. I'll volunteer to make the
new config work with this sites.xml format.


On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Hi,
>
> I committed a fairly large number of changes. Some things got limited
> testing, and some are work in progress. For example, I need to update
> sites files used in tests, but it's getting late here.
> Anyway, here's a summary:
>
> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> things to break, but we discussed cleaning both sites.xml and tc.data
> many times and I guess I never got the courage to do it until now. This
> format is not backwards compatible (intentionally). It goes like this:
> <sites>
>   <apps>? <!-- global apps -->
>     <property name="...">...</property>* <!-- global app properties -->
>     <env name="...">...</env>* <!-- global app envs -->
>     <app name="..." executable="...">*
>        <property/>*
>        <env/>*
>     </app>
>   </apps>
>
>   <site name="...">
>     <execution provider="..."...>
>       <property/>*
>     </execution>
>     <filesystem>?
>      <property/>*
>     </filesystem>
>     <wokdirectory/>
>     <scratch/>?
>     <apps/>?
>   <site>
> </sites>
>
> There are no more namespaces since they were mostly used to figure out
> whether a property was going to the task or the site. I also changed
> "profile" to "property", since "profile" seemed a bit arcane.
>
> You should also now be able to use maxParallelTasks and
> initialParallelTasks instead of jobThrottle and initialScore. It
> computes the latter automatically.
>
> 2. Coasters now support multiple configurations. So you can have two
> local:local coaster sites with different settings and it should work.
> The way this works is that every configuration (and every different run)
> gets its own job queue, settings, and block allocator, and they don't
> interact with each other. This might be problematic with things like
> slots which is meant to limit the number of jobs globally, but that's a
> minor annoyance.
>
> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> default). This will be the jobs per node setting. Each worker can have a
> different number, and that should work as expected. For passive workers,
> the client side jobsPerNode setting will be ignored.
>
> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> an offline version of the http monitor. It can parse logs and feed the
> information to a web browser. It should also be able to follow logs as
> they are produced and feed live information to a browser. It can also
> fake a live log by parsing it slowly instead of all-at-once.
>
> 5. The http monitor got a bit of an update. You can look at all kinds of
> statistics, browse through apps, look at pretty plots of how apps
> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> fashion. It should be cute and useful in one. Won't work with older
> logs. Some pics attached.
>
> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> I/O stats. If you click on a link to a worker in the http monitor,
> you'll get nice graphs of these things. The probes are currently
> hardcoded to run every 60 seconds (although in the pictures below they
> ran at 1/s).
>
> Mihael
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/55fb86b0/attachment.html>

From wilde at anl.gov  Fri Jul  4 18:21:08 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 4 Jul 2014 18:21:08 -0500
Subject: [Swift-devel] changes
In-Reply-To: <CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
Message-ID: <53B736E4.7050202@anl.gov>

David,

Mihael suggested making the new config native to Swift, by converting 
the perl config code into Java.

That sounded pretty reasonable, and would make the format you devised be 
the sole external specification of all properties.

We were thinking that this could be done in 0.96. Thoughts?

- Mike

On 7/4/14, 6:05 PM, David Kelly wrote:
> This looks great - lots of very useful changes. I'll volunteer to make 
> the new config work with this sites.xml format.
>
>
> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov 
> <mailto:hategan at mcs.anl.gov>> wrote:
>
>     Hi,
>
>     I committed a fairly large number of changes. Some things got limited
>     testing, and some are work in progress. For example, I need to update
>     sites files used in tests, but it's getting late here.
>     Anyway, here's a summary:
>
>     1. New sites.xml format and removal of tc.data. I expect a bunch of
>     things to break, but we discussed cleaning both sites.xml and tc.data
>     many times and I guess I never got the courage to do it until now.
>     This
>     format is not backwards compatible (intentionally). It goes like this:
>     <sites>
>       <apps>? <!-- global apps -->
>         <property name="...">...</property>* <!-- global app
>     properties -->
>         <env name="...">...</env>* <!-- global app envs -->
>         <app name="..." executable="...">*
>            <property/>*
>            <env/>*
>         </app>
>       </apps>
>
>       <site name="...">
>         <execution provider="..."...>
>           <property/>*
>         </execution>
>         <filesystem>?
>          <property/>*
>         </filesystem>
>         <wokdirectory/>
>         <scratch/>?
>         <apps/>?
>       <site>
>     </sites>
>
>     There are no more namespaces since they were mostly used to figure out
>     whether a property was going to the task or the site. I also changed
>     "profile" to "property", since "profile" seemed a bit arcane.
>
>     You should also now be able to use maxParallelTasks and
>     initialParallelTasks instead of jobThrottle and initialScore. It
>     computes the latter automatically.
>
>     2. Coasters now support multiple configurations. So you can have two
>     local:local coaster sites with different settings and it should work.
>     The way this works is that every configuration (and every
>     different run)
>     gets its own job queue, settings, and block allocator, and they don't
>     interact with each other. This might be problematic with things like
>     slots which is meant to limit the number of jobs globally, but
>     that's a
>     minor annoyance.
>
>     3. Passive workers can be launched with a -c(oncurrency) argument
>     (1 by
>     default). This will be the jobs per node setting. Each worker can
>     have a
>     different number, and that should work as expected. For passive
>     workers,
>     the client side jobsPerNode setting will be ignored.
>
>     4. There is a tool to analyze logs now (swift-log-info). This is
>     simply
>     an offline version of the http monitor. It can parse logs and feed the
>     information to a web browser. It should also be able to follow logs as
>     they are produced and feed live information to a browser. It can also
>     fake a live log by parsing it slowly instead of all-at-once.
>
>     5. The http monitor got a bit of an update. You can look at all
>     kinds of
>     statistics, browse through apps, look at pretty plots of how apps
>     behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>     fashion. It should be cute and useful in one. Won't work with older
>     logs. Some pics attached.
>
>     6. There are now worker.pl <http://worker.pl> "probes" for CPU
>     usage, disk usage, and some
>     I/O stats. If you click on a link to a worker in the http monitor,
>     you'll get nice graphs of these things. The probes are currently
>     hardcoded to run every 60 seconds (although in the pictures below they
>     ran at 1/s).
>
>     Mihael
>
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/19d7f236/attachment.html>

From davidkelly at uchicago.edu  Fri Jul  4 18:45:19 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Fri, 4 Jul 2014 18:45:19 -0500
Subject: [Swift-devel] changes
In-Reply-To: <53B736E4.7050202@anl.gov>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
Message-ID: <CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>

Great, I think that's a good idea. Most of how it works should be described
in the trunk userguide - how to handle run directories, config syntax,
search paths for swift.properties, templating, etc.


On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:

>  David,
>
> Mihael suggested making the new config native to Swift, by converting the
> perl config code into Java.
>
> That sounded pretty reasonable, and would make the format you devised be
> the sole external specification of all properties.
>
> We were thinking that this could be done in 0.96. Thoughts?
>
> - Mike
>
>
> On 7/4/14, 6:05 PM, David Kelly wrote:
>
> This looks great - lots of very useful changes. I'll volunteer to make the
> new config work with this sites.xml format.
>
>
> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>
>> Hi,
>>
>> I committed a fairly large number of changes. Some things got limited
>> testing, and some are work in progress. For example, I need to update
>> sites files used in tests, but it's getting late here.
>> Anyway, here's a summary:
>>
>> 1. New sites.xml format and removal of tc.data. I expect a bunch of
>> things to break, but we discussed cleaning both sites.xml and tc.data
>> many times and I guess I never got the courage to do it until now. This
>> format is not backwards compatible (intentionally). It goes like this:
>> <sites>
>>   <apps>? <!-- global apps -->
>>     <property name="...">...</property>* <!-- global app properties -->
>>     <env name="...">...</env>* <!-- global app envs -->
>>     <app name="..." executable="...">*
>>        <property/>*
>>        <env/>*
>>     </app>
>>   </apps>
>>
>>   <site name="...">
>>     <execution provider="..."...>
>>       <property/>*
>>     </execution>
>>     <filesystem>?
>>      <property/>*
>>     </filesystem>
>>     <wokdirectory/>
>>     <scratch/>?
>>     <apps/>?
>>   <site>
>> </sites>
>>
>> There are no more namespaces since they were mostly used to figure out
>> whether a property was going to the task or the site. I also changed
>> "profile" to "property", since "profile" seemed a bit arcane.
>>
>> You should also now be able to use maxParallelTasks and
>> initialParallelTasks instead of jobThrottle and initialScore. It
>> computes the latter automatically.
>>
>> 2. Coasters now support multiple configurations. So you can have two
>> local:local coaster sites with different settings and it should work.
>> The way this works is that every configuration (and every different run)
>> gets its own job queue, settings, and block allocator, and they don't
>> interact with each other. This might be problematic with things like
>> slots which is meant to limit the number of jobs globally, but that's a
>> minor annoyance.
>>
>> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
>> default). This will be the jobs per node setting. Each worker can have a
>> different number, and that should work as expected. For passive workers,
>> the client side jobsPerNode setting will be ignored.
>>
>> 4. There is a tool to analyze logs now (swift-log-info). This is simply
>> an offline version of the http monitor. It can parse logs and feed the
>> information to a web browser. It should also be able to follow logs as
>> they are produced and feed live information to a browser. It can also
>> fake a live log by parsing it slowly instead of all-at-once.
>>
>> 5. The http monitor got a bit of an update. You can look at all kinds of
>> statistics, browse through apps, look at pretty plots of how apps
>> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>> fashion. It should be cute and useful in one. Won't work with older
>> logs. Some pics attached.
>>
>> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
>> I/O stats. If you click on a link to a worker in the http monitor,
>> you'll get nice graphs of these things. The probes are currently
>> hardcoded to run every 60 seconds (although in the pictures below they
>> ran at 1/s).
>>
>> Mihael
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>>
>
>
> _______________________________________________
> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> --
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140704/2f067e2f/attachment.html>

From hategan at mcs.anl.gov  Sat Jul  5 00:58:02 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 4 Jul 2014 22:58:02 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
Message-ID: <1404539882.26546.8.camel@echo>

Thanks. I'll probably have questions as I go along.

So when I spoke to Mike earlier I suggested YAML. After reading a bit I
became convinced that YAML is not such a good idea. It's too complex and
ambiguous at times.

The unfortunate thing is that there is no good standardized hierarchical
language for this sort of business except for XML. However, after some
searching I found this: https://github.com/typesafehub/config

It has a format that is a superset of JSON which allows a few nice
syntactic changes that make it more user-friendly, such as not having to
quote keys, the ability to use ":" and "=" interchangeably, property/env
var expansion, and the ability to omit the separating comma if there is
a new line. For example:

foo {
    bar = 10
    baz = 12
}

This is I believe very close to David's format.

But there is something else, which I like, and that is include
directives. I think we ran into a number of problems with the fact that
Swift loads various configurations and that can be surprising to a user.
At the same time, some users do want to have some default settings and
only override a few.

I believe that includes address this problem elegantly in that users can
use common configurations and override a few things, but it will always
be obvious when looking at a config file that there are things coming
from another file.

Mihael

On Fri, 2014-07-04 at 18:45 -0500, David Kelly wrote:
> Great, I think that's a good idea. Most of how it works should be described
> in the trunk userguide - how to handle run directories, config syntax,
> search paths for swift.properties, templating, etc.
> 
> 
> On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:
> 
> >  David,
> >
> > Mihael suggested making the new config native to Swift, by converting the
> > perl config code into Java.
> >
> > That sounded pretty reasonable, and would make the format you devised be
> > the sole external specification of all properties.
> >
> > We were thinking that this could be done in 0.96. Thoughts?
> >
> > - Mike
> >
> >
> > On 7/4/14, 6:05 PM, David Kelly wrote:
> >
> > This looks great - lots of very useful changes. I'll volunteer to make the
> > new config work with this sites.xml format.
> >
> >
> > On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> >
> >> Hi,
> >>
> >> I committed a fairly large number of changes. Some things got limited
> >> testing, and some are work in progress. For example, I need to update
> >> sites files used in tests, but it's getting late here.
> >> Anyway, here's a summary:
> >>
> >> 1. New sites.xml format and removal of tc.data. I expect a bunch of
> >> things to break, but we discussed cleaning both sites.xml and tc.data
> >> many times and I guess I never got the courage to do it until now. This
> >> format is not backwards compatible (intentionally). It goes like this:
> >> <sites>
> >>   <apps>? <!-- global apps -->
> >>     <property name="...">...</property>* <!-- global app properties -->
> >>     <env name="...">...</env>* <!-- global app envs -->
> >>     <app name="..." executable="...">*
> >>        <property/>*
> >>        <env/>*
> >>     </app>
> >>   </apps>
> >>
> >>   <site name="...">
> >>     <execution provider="..."...>
> >>       <property/>*
> >>     </execution>
> >>     <filesystem>?
> >>      <property/>*
> >>     </filesystem>
> >>     <wokdirectory/>
> >>     <scratch/>?
> >>     <apps/>?
> >>   <site>
> >> </sites>
> >>
> >> There are no more namespaces since they were mostly used to figure out
> >> whether a property was going to the task or the site. I also changed
> >> "profile" to "property", since "profile" seemed a bit arcane.
> >>
> >> You should also now be able to use maxParallelTasks and
> >> initialParallelTasks instead of jobThrottle and initialScore. It
> >> computes the latter automatically.
> >>
> >> 2. Coasters now support multiple configurations. So you can have two
> >> local:local coaster sites with different settings and it should work.
> >> The way this works is that every configuration (and every different run)
> >> gets its own job queue, settings, and block allocator, and they don't
> >> interact with each other. This might be problematic with things like
> >> slots which is meant to limit the number of jobs globally, but that's a
> >> minor annoyance.
> >>
> >> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
> >> default). This will be the jobs per node setting. Each worker can have a
> >> different number, and that should work as expected. For passive workers,
> >> the client side jobsPerNode setting will be ignored.
> >>
> >> 4. There is a tool to analyze logs now (swift-log-info). This is simply
> >> an offline version of the http monitor. It can parse logs and feed the
> >> information to a web browser. It should also be able to follow logs as
> >> they are produced and feed live information to a browser. It can also
> >> fake a live log by parsing it slowly instead of all-at-once.
> >>
> >> 5. The http monitor got a bit of an update. You can look at all kinds of
> >> statistics, browse through apps, look at pretty plots of how apps
> >> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
> >> fashion. It should be cute and useful in one. Won't work with older
> >> logs. Some pics attached.
> >>
> >> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
> >> I/O stats. If you click on a link to a worker in the http monitor,
> >> you'll get nice graphs of these things. The probes are currently
> >> hardcoded to run every 60 seconds (although in the pictures below they
> >> ran at 1/s).
> >>
> >> Mihael
> >>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>
> >>
> >
> >
> > _______________________________________________
> > Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > --
> > Michael Wilde
> > Mathematics and Computer Science          Computation Institute
> > Argonne National Laboratory               The University of Chicago
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From wilde at anl.gov  Sat Jul  5 09:34:33 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sat, 5 Jul 2014 09:34:33 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404539882.26546.8.camel@echo>
References: <1404461246.11929.27.camel@echo>	
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>	
	<53B736E4.7050202@anl.gov>	
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo>
Message-ID: <53B80CF9.5000700@anl.gov>

This looks very good, deals with a lot of parsing issues, and is 
licensed under Apache 2 (same as Swift).

But is it Java-only?   If so, then down the road, for Swift/T with 
Coaster C client, just parse options using a Java app perhaps?

- Mike

On 7/5/14, 12:58 AM, Mihael Hategan wrote:
> Thanks. I'll probably have questions as I go along.
>
> So when I spoke to Mike earlier I suggested YAML. After reading a bit I
> became convinced that YAML is not such a good idea. It's too complex and
> ambiguous at times.
>
> The unfortunate thing is that there is no good standardized hierarchical
> language for this sort of business except for XML. However, after some
> searching I found this: https://github.com/typesafehub/config
>
> It has a format that is a superset of JSON which allows a few nice
> syntactic changes that make it more user-friendly, such as not having to
> quote keys, the ability to use ":" and "=" interchangeably, property/env
> var expansion, and the ability to omit the separating comma if there is
> a new line. For example:
>
> foo {
>      bar = 10
>      baz = 12
> }
>
> This is I believe very close to David's format.
>
> But there is something else, which I like, and that is include
> directives. I think we ran into a number of problems with the fact that
> Swift loads various configurations and that can be surprising to a user.
> At the same time, some users do want to have some default settings and
> only override a few.
>
> I believe that includes address this problem elegantly in that users can
> use common configurations and override a few things, but it will always
> be obvious when looking at a config file that there are things coming
> from another file.
>
> Mihael
>
> On Fri, 2014-07-04 at 18:45 -0500, David Kelly wrote:
>> Great, I think that's a good idea. Most of how it works should be described
>> in the trunk userguide - how to handle run directories, config syntax,
>> search paths for swift.properties, templating, etc.
>>
>>
>> On Fri, Jul 4, 2014 at 6:21 PM, Michael Wilde <wilde at anl.gov> wrote:
>>
>>>   David,
>>>
>>> Mihael suggested making the new config native to Swift, by converting the
>>> perl config code into Java.
>>>
>>> That sounded pretty reasonable, and would make the format you devised be
>>> the sole external specification of all properties.
>>>
>>> We were thinking that this could be done in 0.96. Thoughts?
>>>
>>> - Mike
>>>
>>>
>>> On 7/4/14, 6:05 PM, David Kelly wrote:
>>>
>>> This looks great - lots of very useful changes. I'll volunteer to make the
>>> new config work with this sites.xml format.
>>>
>>>
>>> On Fri, Jul 4, 2014 at 3:07 AM, Mihael Hategan <hategan at mcs.anl.gov>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I committed a fairly large number of changes. Some things got limited
>>>> testing, and some are work in progress. For example, I need to update
>>>> sites files used in tests, but it's getting late here.
>>>> Anyway, here's a summary:
>>>>
>>>> 1. New sites.xml format and removal of tc.data. I expect a bunch of
>>>> things to break, but we discussed cleaning both sites.xml and tc.data
>>>> many times and I guess I never got the courage to do it until now. This
>>>> format is not backwards compatible (intentionally). It goes like this:
>>>> <sites>
>>>>    <apps>? <!-- global apps -->
>>>>      <property name="...">...</property>* <!-- global app properties -->
>>>>      <env name="...">...</env>* <!-- global app envs -->
>>>>      <app name="..." executable="...">*
>>>>         <property/>*
>>>>         <env/>*
>>>>      </app>
>>>>    </apps>
>>>>
>>>>    <site name="...">
>>>>      <execution provider="..."...>
>>>>        <property/>*
>>>>      </execution>
>>>>      <filesystem>?
>>>>       <property/>*
>>>>      </filesystem>
>>>>      <wokdirectory/>
>>>>      <scratch/>?
>>>>      <apps/>?
>>>>    <site>
>>>> </sites>
>>>>
>>>> There are no more namespaces since they were mostly used to figure out
>>>> whether a property was going to the task or the site. I also changed
>>>> "profile" to "property", since "profile" seemed a bit arcane.
>>>>
>>>> You should also now be able to use maxParallelTasks and
>>>> initialParallelTasks instead of jobThrottle and initialScore. It
>>>> computes the latter automatically.
>>>>
>>>> 2. Coasters now support multiple configurations. So you can have two
>>>> local:local coaster sites with different settings and it should work.
>>>> The way this works is that every configuration (and every different run)
>>>> gets its own job queue, settings, and block allocator, and they don't
>>>> interact with each other. This might be problematic with things like
>>>> slots which is meant to limit the number of jobs globally, but that's a
>>>> minor annoyance.
>>>>
>>>> 3. Passive workers can be launched with a -c(oncurrency) argument (1 by
>>>> default). This will be the jobs per node setting. Each worker can have a
>>>> different number, and that should work as expected. For passive workers,
>>>> the client side jobsPerNode setting will be ignored.
>>>>
>>>> 4. There is a tool to analyze logs now (swift-log-info). This is simply
>>>> an offline version of the http monitor. It can parse logs and feed the
>>>> information to a web browser. It should also be able to follow logs as
>>>> they are produced and feed live information to a browser. It can also
>>>> fake a live log by parsing it slowly instead of all-at-once.
>>>>
>>>> 5. The http monitor got a bit of an update. You can look at all kinds of
>>>> statistics, browse through apps, look at pretty plots of how apps
>>>> behaved, etc. all in a nice dynamically updated ajaxy and hyperlinked
>>>> fashion. It should be cute and useful in one. Won't work with older
>>>> logs. Some pics attached.
>>>>
>>>> 6. There are now worker.pl "probes" for CPU usage, disk usage, and some
>>>> I/O stats. If you click on a link to a worker in the http monitor,
>>>> you'll get nice graphs of these things. The probes are currently
>>>> hardcoded to run every 60 seconds (although in the pictures below they
>>>> ran at 1/s).
>>>>
>>>> Mihael
>>>>
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Swift-devel mailing listSwift-devel at ci.uchicago.eduhttps://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>>> --
>>> Michael Wilde
>>> Mathematics and Computer Science          Computation Institute
>>> Argonne National Laboratory               The University of Chicago
>>>
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Sat Jul  5 13:03:30 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 5 Jul 2014 11:03:30 -0700
Subject: [Swift-devel] changes
In-Reply-To: <53B80CF9.5000700@anl.gov>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
Message-ID: <1404583410.32736.2.camel@echo>

On Sat, 2014-07-05 at 09:34 -0500, Michael Wilde wrote:
> This looks very good, deals with a lot of parsing issues, and is 
> licensed under Apache 2 (same as Swift).
> 
> But is it Java-only?   If so, then down the road, for Swift/T with 
> Coaster C client, just parse options using a Java app perhaps?

It's Java only. I don't know. I guess writing one parser is better than
writing two parsers.

The only other reasonable choice is XML unless somebody knows something
else.

Mihael




From wilde at anl.gov  Sat Jul  5 13:11:16 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sat, 5 Jul 2014 13:11:16 -0500
Subject: [Swift-devel] New parsing tool for swift.properties
In-Reply-To: <1404583410.32736.2.camel@echo>
References: <1404461246.11929.27.camel@echo>		
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>		
	<53B736E4.7050202@anl.gov>		
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>	
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
Message-ID: <53B83FC4.90408@anl.gov>

was: Re: [Swift-devel] changes

I think this tool (Typesafe config) looks promising; I would try it.

Once you have a property parser in Swift/K, that can be readily turned 
into a standalone Java command to run for Swift/T, if/when needed, which 
can emit Tcl or whatever other flat propoerty format is needed.

Sound reasonable?

- Mike


On 7/5/14, 1:03 PM, Mihael Hategan wrote:
> On Sat, 2014-07-05 at 09:34 -0500, Michael Wilde wrote:
>> This looks very good, deals with a lot of parsing issues, and is
>> licensed under Apache 2 (same as Swift).
>>
>> But is it Java-only?   If so, then down the road, for Swift/T with
>> Coaster C client, just parse options using a Java app perhaps?
> It's Java only. I don't know. I guess writing one parser is better than
> writing two parsers.
>
> The only other reasonable choice is XML unless somebody knows something
> else.
>
> Mihael
>
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Sat Jul  5 13:20:21 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 5 Jul 2014 11:20:21 -0700
Subject: [Swift-devel] New parsing tool for swift.properties
In-Reply-To: <53B83FC4.90408@anl.gov>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo> <53B83FC4.90408@anl.gov>
Message-ID: <1404584421.393.5.camel@echo>

Maybe. I have no idea how the /K configuration structure makes sense
in /T. Tim, Justin?

Mihael

On Sat, 2014-07-05 at 13:11 -0500, Michael Wilde wrote:
> was: Re: [Swift-devel] changes
> 
> I think this tool (Typesafe config) looks promising; I would try it.
> 
> Once you have a property parser in Swift/K, that can be readily turned 
> into a standalone Java command to run for Swift/T, if/when needed, which 
> can emit Tcl or whatever other flat propoerty format is needed.
> 
> Sound reasonable?
> 
> - Mike
> 
> 
> On 7/5/14, 1:03 PM, Mihael Hategan wrote:
> > On Sat, 2014-07-05 at 09:34 -0500, Michael Wilde wrote:
> >> This looks very good, deals with a lot of parsing issues, and is
> >> licensed under Apache 2 (same as Swift).
> >>
> >> But is it Java-only?   If so, then down the road, for Swift/T with
> >> Coaster C client, just parse options using a Java app perhaps?
> > It's Java only. I don't know. I guess writing one parser is better than
> > writing two parsers.
> >
> > The only other reasonable choice is XML unless somebody knows something
> > else.
> >
> > Mihael
> >
> >
> 




From tim.g.armstrong at gmail.com  Sat Jul  5 20:28:17 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sat, 5 Jul 2014 20:28:17 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404583410.32736.2.camel@echo>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
Message-ID: <CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>

Trying to run a java parser from the C client sounds like an additional
source of deployment/configuration hassles.  Is there a particular reason
why it needs to be the client though rather than the Coaster service that
handles the configuration?

To me it would make the most sense to either choose a simple constrained
format that its feasible to write a custom parser for, or to just use JSON
or XML where there are lots of existing parsers.

- Tim


On Sat, Jul 5, 2014 at 1:03 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sat, 2014-07-05 at 09:34 -0500, Michael Wilde wrote:
> > This looks very good, deals with a lot of parsing issues, and is
> > licensed under Apache 2 (same as Swift).
> >
> > But is it Java-only?   If so, then down the road, for Swift/T with
> > Coaster C client, just parse options using a Java app perhaps?
>
> It's Java only. I don't know. I guess writing one parser is better than
> writing two parsers.
>
> The only other reasonable choice is XML unless somebody knows something
> else.
>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140705/fad357d3/attachment.html>

From tim.g.armstrong at gmail.com  Sat Jul  5 20:37:06 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sat, 5 Jul 2014 20:37:06 -0500
Subject: [Swift-devel] New parsing tool for swift.properties
In-Reply-To: <1404584421.393.5.camel@echo>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo> <53B83FC4.90408@anl.gov>
	<1404584421.393.5.camel@echo>
Message-ID: <CAC0jiV6xtrY6mkW3mpHdGV9ixDipxrQpyPDjw86C43=6ZWcY1A@mail.gmail.com>

I already replied to the other email but I'd strongly argue for a solution
that avoids imposing any additional steps, intermediate files, or
dependencies in running a Swift/T program.  It's difficult enough to
compile and run a Swift/T application already.


On Sat, Jul 5, 2014 at 1:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Maybe. I have no idea how the /K configuration structure makes sense
> in /T. Tim, Justin?
>
> Mihael
>
> On Sat, 2014-07-05 at 13:11 -0500, Michael Wilde wrote:
> > was: Re: [Swift-devel] changes
> >
> > I think this tool (Typesafe config) looks promising; I would try it.
> >
> > Once you have a property parser in Swift/K, that can be readily turned
> > into a standalone Java command to run for Swift/T, if/when needed, which
> > can emit Tcl or whatever other flat propoerty format is needed.
> >
> > Sound reasonable?
> >
> > - Mike
> >
> >
> > On 7/5/14, 1:03 PM, Mihael Hategan wrote:
> > > On Sat, 2014-07-05 at 09:34 -0500, Michael Wilde wrote:
> > >> This looks very good, deals with a lot of parsing issues, and is
> > >> licensed under Apache 2 (same as Swift).
> > >>
> > >> But is it Java-only?   If so, then down the road, for Swift/T with
> > >> Coaster C client, just parse options using a Java app perhaps?
> > > It's Java only. I don't know. I guess writing one parser is better than
> > > writing two parsers.
> > >
> > > The only other reasonable choice is XML unless somebody knows something
> > > else.
> > >
> > > Mihael
> > >
> > >
> >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140705/d98ab1b8/attachment.html>

From hategan at mcs.anl.gov  Sat Jul  5 21:16:48 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 5 Jul 2014 19:16:48 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
Message-ID: <1404613008.8643.20.camel@echo>

On Sat, 2014-07-05 at 20:28 -0500, Tim Armstrong wrote:
> Trying to run a java parser from the C client sounds like an additional
> source of deployment/configuration hassles.  Is there a particular reason
> why it needs to be the client though rather than the Coaster service that
> handles the configuration?

It's not just coasters. This applies to all of Swift/K, and includes all
the options, site and app definitions. Coasters are a small part, and
stand-alone coasters where the service would have a single static
configuration is an even smaller part.

> 
> To me it would make the most sense to either choose a simple constrained
> format that its feasible to write a custom parser for, or to just use JSON
> or XML where there are lots of existing parsers.

Yeah, and there's the problem.
We have a new config mechanism that Mike and David developed. It's very
nice to the eyes. It's supposed to be the next configuration format for
swift. It's almost JSON, except for a few syntactic sugary things.
JSON's habit of quoting every key is noisy, probably more so than XML.

I'm taking that format and using a pre-existing library to do the
parsing and skipping intermediate XML files.

We can argue a few points:
- do we really want this new format given that it's non-standard and
there is some unclear as of yet need to have something maybe similar in
Swift/T?
- if we do, do we want it to be implemented as a custom perl translator
to XML and then a Java XML parser, or a Java direct parser?
- do Swift/T and K really need to have the same configuration format
when they differ in so many other respects?

Mihael



From tim.g.armstrong at gmail.com  Sun Jul  6 15:01:07 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sun, 6 Jul 2014 15:01:07 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404613008.8643.20.camel@echo>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
Message-ID: <CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>

I think the goal would be to allow the same sites files to be used for
Swift/K and Swift/T+coasters, right? Currently Swift/T is mainly configured
through environment variables.

It seems like it might make the most sense architecturally in Swift/T for
the Coaster service to load the config file directly, given that the config
file will probably just be slurped up and sent to the service anyway.  If
this was possible it might avoid the issue entirely.

The main issue with using a parser that accepts a superset of the official
config file syntax is that config files with syntax that "officially" is
invalid will probably crop up.  In practice this might be a relatively
small problem but I can see users being confused by it.

I do think a custom format would need more documentation than is currently
in the user guide - there's no mention of how whitespace is handled, for
example.

My two cents is that just parsing it directly from Java will save effort in
the long run and improve the user experience with better error reporting
etc.  Having multiple stages of processing also makes it harder to ensure
that special characters, whitespace, etc is correctly preserved, and makes
it hard to e.g. provide line numbers in error messages if desired.

- Tim


On Sat, Jul 5, 2014 at 9:16 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sat, 2014-07-05 at 20:28 -0500, Tim Armstrong wrote:
> > Trying to run a java parser from the C client sounds like an additional
> > source of deployment/configuration hassles.  Is there a particular reason
> > why it needs to be the client though rather than the Coaster service that
> > handles the configuration?
>
> It's not just coasters. This applies to all of Swift/K, and includes all
> the options, site and app definitions. Coasters are a small part, and
> stand-alone coasters where the service would have a single static
> configuration is an even smaller part.
>
> >
> > To me it would make the most sense to either choose a simple constrained
> > format that its feasible to write a custom parser for, or to just use
> JSON
> > or XML where there are lots of existing parsers.
>
> Yeah, and there's the problem.
> We have a new config mechanism that Mike and David developed. It's very
> nice to the eyes. It's supposed to be the next configuration format for
> swift. It's almost JSON, except for a few syntactic sugary things.
> JSON's habit of quoting every key is noisy, probably more so than XML.
>
> I'm taking that format and using a pre-existing library to do the
> parsing and skipping intermediate XML files.
>
> We can argue a few points:
> - do we really want this new format given that it's non-standard and
> there is some unclear as of yet need to have something maybe similar in
> Swift/T?
> - if we do, do we want it to be implemented as a custom perl translator
> to XML and then a Java XML parser, or a Java direct parser?
> - do Swift/T and K really need to have the same configuration format
> when they differ in so many other respects?
>
> Mihael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140706/4285ed0b/attachment.html>

From hategan at mcs.anl.gov  Sun Jul  6 15:55:55 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 6 Jul 2014 13:55:55 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
	<CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
Message-ID: <1404680155.10983.19.camel@echo>

On Sun, 2014-07-06 at 15:01 -0500, Tim Armstrong wrote:
> I think the goal would be to allow the same sites files to be used for
> Swift/K and Swift/T+coasters, right? Currently Swift/T is mainly configured
> through environment variables.
> 
> It seems like it might make the most sense architecturally in Swift/T for
> the Coaster service to load the config file directly, given that the config
> file will probably just be slurped up and sent to the service anyway.  If
> this was possible it might avoid the issue entirely.

Currently the configuration comes from the client. There is a plan to
have a version of the coaster service where the service is started with
an immutable configuration. This is a relatively different issue. It
does not need a hierarchical configuration format, and it will most
likely be a flat properties file. So if that addresses the problem
from /T's perspective, then things should be fine.

> 
> The main issue with using a parser that accepts a superset of the official
> config file syntax is that config files with syntax that "officially" is
> invalid will probably crop up.  In practice this might be a relatively
> small problem but I can see users being confused by it.

The choice is not that. The choice at the moment is between an in-house
format that isn't well documented, does not specify how special
characters are handled (or expressed), does not escape much (try & in an
environment variable) etc. and a variant of a standard that isn't
completely in-house and that handles the corner cases. Well, that or
back to XML.

> 
> I do think a custom format would need more documentation than is currently
> in the user guide - there's no mention of how whitespace is handled, for
> example.

I think everybody agrees that the current documentation isn't quite
right. Fixing that would be part of the change.

> 
> My two cents is that just parsing it directly from Java will save effort in
> the long run and improve the user experience with better error reporting
> etc.  Having multiple stages of processing also makes it harder to ensure
> that special characters, whitespace, etc is correctly preserved, and makes
> it hard to e.g. provide line numbers in error messages if desired.

Right. Hence my choice of doing this. An additional difficulty was that
it seemed hard to convince the current translator to accommodate a
more-than-one-nesting-level hierarchical structure.

There are more benefits, in that there would now be a unified validation
place that checks all configuration values and fails early. This was
previously either missing (for tc.data) or scattered in different places
(providers for sites.xml, the perl code for the sites translator, custom
java code for the other swift properties).

Mihael



From tim.g.armstrong at gmail.com  Sun Jul  6 17:20:45 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sun, 6 Jul 2014 17:20:45 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404680155.10983.19.camel@echo>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
	<CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
	<1404680155.10983.19.camel@echo>
Message-ID: <CAC0jiV7sDA6RizFcKjMvQGk6TR21c2iySZZ4RAFGCY7esO7WkA@mail.gmail.com>

> Currently the configuration comes from the client. There is a plan to
> have a version of the coaster service where the service is started with
> an immutable configuration. This is a relatively different issue. It
> does not need a hierarchical configuration format, and it will most
> likely be a flat properties file. So if that addresses the problem
> from /T's perspective, then things should be fine.

I don't think I understand well enough what the pros/cons are here.  With
the static configuration, we'd lose the ability for the client to
dynamically add new sites or modify other settings - is that the only
difference?  I understand that it maybe buys you some flexibility in that
you can stand up a coasters service ahead of time, or reuse the same
service for Swift runs with different configurations. I don't know if those
are likely use cases for Swift/T.  Are there any Coasters settings where
there's some other reason the client needs to set it?

- Tim



On Sun, Jul 6, 2014 at 3:55 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2014-07-06 at 15:01 -0500, Tim Armstrong wrote:
> > I think the goal would be to allow the same sites files to be used for
> > Swift/K and Swift/T+coasters, right? Currently Swift/T is mainly
> configured
> > through environment variables.
> >
> > It seems like it might make the most sense architecturally in Swift/T for
> > the Coaster service to load the config file directly, given that the
> config
> > file will probably just be slurped up and sent to the service anyway.  If
> > this was possible it might avoid the issue entirely.
>
> Currently the configuration comes from the client. There is a plan to
> have a version of the coaster service where the service is started with
> an immutable configuration. This is a relatively different issue. It
> does not need a hierarchical configuration format, and it will most
> likely be a flat properties file. So if that addresses the problem
> from /T's perspective, then things should be fine.
>
> >
> > The main issue with using a parser that accepts a superset of the
> official
> > config file syntax is that config files with syntax that "officially" is
> > invalid will probably crop up.  In practice this might be a relatively
> > small problem but I can see users being confused by it.
>
> The choice is not that. The choice at the moment is between an in-house
> format that isn't well documented, does not specify how special
> characters are handled (or expressed), does not escape much (try & in an
> environment variable) etc. and a variant of a standard that isn't
> completely in-house and that handles the corner cases. Well, that or
> back to XML.
>
> >
> > I do think a custom format would need more documentation than is
> currently
> > in the user guide - there's no mention of how whitespace is handled, for
> > example.
>
> I think everybody agrees that the current documentation isn't quite
> right. Fixing that would be part of the change.
>
> >
> > My two cents is that just parsing it directly from Java will save effort
> in
> > the long run and improve the user experience with better error reporting
> > etc.  Having multiple stages of processing also makes it harder to ensure
> > that special characters, whitespace, etc is correctly preserved, and
> makes
> > it hard to e.g. provide line numbers in error messages if desired.
>
> Right. Hence my choice of doing this. An additional difficulty was that
> it seemed hard to convince the current translator to accommodate a
> more-than-one-nesting-level hierarchical structure.
>
> There are more benefits, in that there would now be a unified validation
> place that checks all configuration values and fails early. This was
> previously either missing (for tc.data) or scattered in different places
> (providers for sites.xml, the perl code for the sites translator, custom
> java code for the other swift properties).
>
> Mihael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140706/6be02a19/attachment.html>

From hategan at mcs.anl.gov  Sun Jul  6 17:47:16 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 6 Jul 2014 15:47:16 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV7sDA6RizFcKjMvQGk6TR21c2iySZZ4RAFGCY7esO7WkA@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
	<CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
	<1404680155.10983.19.camel@echo>
	<CAC0jiV7sDA6RizFcKjMvQGk6TR21c2iySZZ4RAFGCY7esO7WkA@mail.gmail.com>
Message-ID: <1404686836.12621.15.camel@echo>

On Sun, 2014-07-06 at 17:20 -0500, Tim Armstrong wrote:
> > Currently the configuration comes from the client. There is a plan to
> > have a version of the coaster service where the service is started with
> > an immutable configuration. This is a relatively different issue. It
> > does not need a hierarchical configuration format, and it will most
> > likely be a flat properties file. So if that addresses the problem
> > from /T's perspective, then things should be fine.
> 
> I don't think I understand well enough what the pros/cons are here.  With
> the static configuration, we'd lose the ability for the client to
> dynamically add new sites

Adding a site, in the sense of a new machine, won't affect services on
other machines. So this would be a separate issue. Think of coasters as
a faster GRAM. There is no provision for automatically managing a pool
of clusters in there.

>  or modify other settings

Right. You would lose the ability to modify settings at will from the
client.

>  - is that the only
> difference?  I understand that it maybe buys you some flexibility in that
> you can stand up a coasters service ahead of time, or reuse the same
> service for Swift runs with different configurations. I don't know if those
> are likely use cases for Swift/T.

I think standing up services ahead of time is the only possibility right
now, since the C client does not automatically start services.

>   Are there any Coasters settings where
> there's some other reason the client needs to set it?

Not that I can think of.

To summarize why I replied previously. You mentioned the coaster service
parsing its configuration. I'm stating that it's impossible to get both
flexibility and service-side parsing.

If the service is to load the configuration, then he client only
controls configuration to the extent that it could make a choices
between a number of pre-set configurations.

So if the service loads the configuration, that is less flexible than
when the client is able to tweak all the knobs. And for the client to be
able to do that based on some user specification, it needs to be able to
read that user specification. 

The bottom line is either server side parsing with the client having
limited control or client side parsing and that needs the client to be
able parse.

Mihael



From tim.g.armstrong at gmail.com  Sun Jul  6 20:35:43 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sun, 6 Jul 2014 20:35:43 -0500
Subject: [Swift-devel] changes
In-Reply-To: <1404686836.12621.15.camel@echo>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
	<CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
	<1404680155.10983.19.camel@echo>
	<CAC0jiV7sDA6RizFcKjMvQGk6TR21c2iySZZ4RAFGCY7esO7WkA@mail.gmail.com>
	<1404686836.12621.15.camel@echo>
Message-ID: <CAC0jiV6FAd0JNJWZ+4npzgN9x5qaEYZR9Z5_Bc3cf7n8-ysncA@mail.gmail.com>

Right, I get that it's more flexible: I was mainly trying to understand how
important that flexibility was in practice.

The main use case I think would be to start a Coaster service at the same
time as the Swift/T run and tear it down once the run was done: e.g. with a
launch script that starts both the Coaster service and the Swift run.  It
sounds like starting a Coaster service pointed at a configuration file
would achieve exactly the same thing as starting a Coaster service then
configuring it via the client in this scenario.  I don't know the config
mechanism and settings well enough to be confident about that assessment is
my problem.

Anyway, this is fairly academic at the moment :)

- Tim


On Sun, Jul 6, 2014 at 5:47 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2014-07-06 at 17:20 -0500, Tim Armstrong wrote:
> > > Currently the configuration comes from the client. There is a plan to
> > > have a version of the coaster service where the service is started with
> > > an immutable configuration. This is a relatively different issue. It
> > > does not need a hierarchical configuration format, and it will most
> > > likely be a flat properties file. So if that addresses the problem
> > > from /T's perspective, then things should be fine.
> >
> > I don't think I understand well enough what the pros/cons are here.  With
> > the static configuration, we'd lose the ability for the client to
> > dynamically add new sites
>
> Adding a site, in the sense of a new machine, won't affect services on
> other machines. So this would be a separate issue. Think of coasters as
> a faster GRAM. There is no provision for automatically managing a pool
> of clusters in there.
>
> >  or modify other settings
>
> Right. You would lose the ability to modify settings at will from the
> client.
>
> >  - is that the only
> > difference?  I understand that it maybe buys you some flexibility in that
> > you can stand up a coasters service ahead of time, or reuse the same
> > service for Swift runs with different configurations. I don't know if
> those
> > are likely use cases for Swift/T.
>
> I think standing up services ahead of time is the only possibility right
> now, since the C client does not automatically start services.
>
> >   Are there any Coasters settings where
> > there's some other reason the client needs to set it?
>
> Not that I can think of.
>
> To summarize why I replied previously. You mentioned the coaster service
> parsing its configuration. I'm stating that it's impossible to get both
> flexibility and service-side parsing.
>
> If the service is to load the configuration, then he client only
> controls configuration to the extent that it could make a choices
> between a number of pre-set configurations.
>
> So if the service loads the configuration, that is less flexible than
> when the client is able to tweak all the knobs. And for the client to be
> able to do that based on some user specification, it needs to be able to
> read that user specification.
>
> The bottom line is either server side parsing with the client having
> limited control or client side parsing and that needs the client to be
> able parse.
>
> Mihael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140706/94c28188/attachment.html>

From hategan at mcs.anl.gov  Sun Jul  6 20:49:13 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 6 Jul 2014 18:49:13 -0700
Subject: [Swift-devel] changes
In-Reply-To: <CAC0jiV6FAd0JNJWZ+4npzgN9x5qaEYZR9Z5_Bc3cf7n8-ysncA@mail.gmail.com>
References: <1404461246.11929.27.camel@echo>
	<CA+_+Ey-GbQwaKX4PRFp-xHndfAp9jebJmsNMJvz4Uw4jS1nqmA@mail.gmail.com>
	<53B736E4.7050202@anl.gov>
	<CA+_+Ey9j6iOc5M=ReWEggzeC=p8wKCFfJnzRCFkAEGrfh19tGA@mail.gmail.com>
	<1404539882.26546.8.camel@echo> <53B80CF9.5000700@anl.gov>
	<1404583410.32736.2.camel@echo>
	<CAC0jiV6_NmvLshOO+2sF3oQgNgEtBM3UgMvTi10oCAJ05i5x_g@mail.gmail.com>
	<1404613008.8643.20.camel@echo>
	<CAC0jiV7Nb-ND_u4mDpZZuWo3JosPrqTHkbQ6Jom9RdnHW8CBww@mail.gmail.com>
	<1404680155.10983.19.camel@echo>
	<CAC0jiV7sDA6RizFcKjMvQGk6TR21c2iySZZ4RAFGCY7esO7WkA@mail.gmail.com>
	<1404686836.12621.15.camel@echo>
	<CAC0jiV6FAd0JNJWZ+4npzgN9x5qaEYZR9Z5_Bc3cf7n8-ysncA@mail.gmail.com>
Message-ID: <1404697753.18341.8.camel@echo>

On Sun, 2014-07-06 at 20:35 -0500, Tim Armstrong wrote:
> Right, I get that it's more flexible: I was mainly trying to understand how
> important that flexibility was in practice.
> 
> The main use case I think would be to start a Coaster service at the same
> time as the Swift/T run and tear it down once the run was done: e.g. with a
> launch script that starts both the Coaster service and the Swift run.  It
> sounds like starting a Coaster service pointed at a configuration file
> would achieve exactly the same thing as starting a Coaster service then
> configuring it via the client in this scenario.

Right. If you don't need multiple configurations per service (which is
only a recent practical development), and if you control the starting of
the service, then it's very reasonable to have the settings loaded by
the service.

But I must point out that this is still a somewhat separate issue. Swift
aggregates configurations for multiple such services. It is not (and
should not) be the duty of a service to understand the entire swift
configuration mechanism and extract the relevant pieces out of it,
though given that the service is Java, some custom launcher could do
something like that while re-using whatever swift uses for
configuration.

>   I don't know the config
> mechanism and settings well enough to be confident about that assessment is
> my problem.

I think there are many possibilities here and that makes it a bit murky.

> 
> Anyway, this is fairly academic at the moment :)

Yeah. I don't see this issue as a blocker. Let's talk details when we
have some tangible goal.

Mihael



From hategan at mcs.anl.gov  Wed Jul  9 15:16:01 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 9 Jul 2014 13:16:01 -0700
Subject: [Swift-devel] new config
Message-ID: <1404936961.26670.15.camel@echo>

Hi,

I committed the new config code. Since it's a pretty big change, it
might take some testing to get it ironed out and some time to get used
to. Particularly, the tests have not been updated. Yadu, when you have
some time, let's work on updating our testing tools.

A bit of an overview:
- it's a JSON-like format, with optional key quoting, optional comma
between object keys (if there is a new line), optional "=" before a "{"
and ":" and "=" mean the same thing. So this is valid:
  site.local {
    execution {
      type: "ssh"
      URL: "localhost"
    }
  }
as is
  site.local.execution.type = "ssh"
  site.local.execution.URL = "localhost"

However, in order to reduce confusion, we should encourage only the
former choice.

- Most properties that were in swift.properties had their names changed
from a dot separator to camelCase. It was distasteful to see a file
where site options were camelCase and other options
used.dots.as.separators.

- Some options have changed a bit more than that. For example,
sitedir.keep is now keepSiteDir.

- I removed clustering (it wasn't used and is superseded by coasters)

- I also removed the graph generation option. It wasn't being used much
(outside of I2U2) and I would prefer that such tools be outside of the
run-time code and in the log analysis tools.

- Everything is validated. Unrecognized options, or invalid values will
produce an error. The "schema" used for validation is in
resources/swift.conf.schema.

- The preferred way of mixing configurations is through includes.

- There is a conversion tool, swift-convert-config. You can give it a
sites.xml, tc.data, and swift.properties, and it will print out a new
config from there.

- Somewhat unrelated, but wrapper/provider staging are now selectable
per site. You would say something like site.local { staging:
"provider" }

- There is also a swift-config-info tool. This is similar to David's
-listconfig option. It can list all available configuration properties
and also print documentation on selected options. For somebody who is
working on the command line doing stuff, it may be more convenient than
switching to a web browser to look up the configuration documentation.

- I still need to finish all property descriptions and update the
documentation.

Mihael



From wilde at anl.gov  Wed Jul  9 15:18:55 2014
From: wilde at anl.gov (Michael Wilde)
Date: Wed, 9 Jul 2014 15:18:55 -0500
Subject: [Swift-devel] new config
In-Reply-To: <1404936961.26670.15.camel@echo>
References: <1404936961.26670.15.camel@echo>
Message-ID: <53BDA3AF.9000504@anl.gov>

Nice step forward!

Do property values need to be quoted? If so, can that be made optional?

- Mike

On 7/9/14, 3:16 PM, Mihael Hategan wrote:
> Hi,
>
> I committed the new config code. Since it's a pretty big change, it
> might take some testing to get it ironed out and some time to get used
> to. Particularly, the tests have not been updated. Yadu, when you have
> some time, let's work on updating our testing tools.
>
> A bit of an overview:
> - it's a JSON-like format, with optional key quoting, optional comma
> between object keys (if there is a new line), optional "=" before a "{"
> and ":" and "=" mean the same thing. So this is valid:
>    site.local {
>      execution {
>        type: "ssh"
>        URL: "localhost"
>      }
>    }
> as is
>    site.local.execution.type = "ssh"
>    site.local.execution.URL = "localhost"
>
> However, in order to reduce confusion, we should encourage only the
> former choice.
>
> - Most properties that were in swift.properties had their names changed
> from a dot separator to camelCase. It was distasteful to see a file
> where site options were camelCase and other options
> used.dots.as.separators.
>
> - Some options have changed a bit more than that. For example,
> sitedir.keep is now keepSiteDir.
>
> - I removed clustering (it wasn't used and is superseded by coasters)
>
> - I also removed the graph generation option. It wasn't being used much
> (outside of I2U2) and I would prefer that such tools be outside of the
> run-time code and in the log analysis tools.
>
> - Everything is validated. Unrecognized options, or invalid values will
> produce an error. The "schema" used for validation is in
> resources/swift.conf.schema.
>
> - The preferred way of mixing configurations is through includes.
>
> - There is a conversion tool, swift-convert-config. You can give it a
> sites.xml, tc.data, and swift.properties, and it will print out a new
> config from there.
>
> - Somewhat unrelated, but wrapper/provider staging are now selectable
> per site. You would say something like site.local { staging:
> "provider" }
>
> - There is also a swift-config-info tool. This is similar to David's
> -listconfig option. It can list all available configuration properties
> and also print documentation on selected options. For somebody who is
> working on the command line doing stuff, it may be more convenient than
> switching to a web browser to look up the configuration documentation.
>
> - I still need to finish all property descriptions and update the
> documentation.
>
> Mihael
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Wed Jul  9 15:23:37 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 9 Jul 2014 13:23:37 -0700
Subject: [Swift-devel] new config
In-Reply-To: <53BDA3AF.9000504@anl.gov>
References: <1404936961.26670.15.camel@echo> <53BDA3AF.9000504@anl.gov>
Message-ID: <1404937417.26906.2.camel@echo>

On Wed, 2014-07-09 at 15:18 -0500, Michael Wilde wrote:
> Nice step forward!
> 
> Do property values need to be quoted?

Not unless they contain special characters, and they must not be quoted
for numeric values and booleans (or lists - notably sites = ["site1",
"site2", ..etc]). However, I would encourage quoting strings. It makes
it more obvious that it's a string rather than some special construct.

Mihael



From iraicu at cs.iit.edu  Thu Jul 10 21:10:27 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Thu, 10 Jul 2014 21:10:27 -0500
Subject: [Swift-devel] CFP: 7th IEEE Workshop on Many-Task Computing on
 Clouds, Grids, and Supercomputers (MTAGS) 2014 @ IEEE/ACM SC14
Message-ID: <53BF4793.7030304@cs.iit.edu>

Call for Papers

---------------------------------------------------------------------------------------
The 7th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2014
http://datasys.cs.iit.edu/events/MTAGS14/
---------------------------------------------------------------------------------------
November 16th, 2014
New Orleans, Louisiana, USA

Co-located with with IEEE/ACM International Conference for
High Performance Computing, Networking, Storage and Analysis (SC14)
In cooperation with ACM SIGHPC

=======================================================================================
The 7th workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) will provide the scientific community a dedicated forum for presenting new research, development, and deployment efforts of large-scale many-task computing (MTC) applications on large scale clusters, Grids, Supercomputers, and Cloud Computing infrastructure. MTC, the theme of the workshop encompasses loosely coupled applications, which are generally composed of many tasks (both independent and dependent tasks) to achieve some larger application goal.  This workshop will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions on all theoretical, simulations, and systems topics related to MTC, but we give special consideration to papers addressing petascale to exascale challenges. Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (pending approval). The workshop will be co-located with the IEEE/ACM Supercomputing 2014 Conference in New Orleans on November 17th, 2014. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/.

For more information on past workshops, please see MTAGS13, MTAGS12, MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. We, the workshop organizers, also published a highly relevant paper that defines Many-Task Computing which was published in MTAGS08, titled ?Many-Task Computing for Grids and Supercomputers?; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing.


Topics
---------------------------------------------------------------------------------------
We invite the submission of original work that is related to the topics below. The papers should be 6 pages, including all figures and references. We aim to cover topics related to Many-Task Computing on each of the three major distributed systems paradigms, Cloud Computing, Grid Computing and Supercomputing. Topics of interest include:
* Compute Resource Management
   * Scheduling
   * Job execution frameworks
   * Local resource manager extensions
   * Performance evaluation of resource managers in use on large scale systems
   * Dynamic resource provisioning
   * Techniques to manage many-core resources and/or GPUs
   * Challenges and opportunities in running many-task workloads on HPC systems
   * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure
* Storage architectures and implementations
   * Distributed file systems
   * Parallel file systems
   * Distributed meta-data management
   * Content distribution systems for large data
   * Data caching frameworks and techniques
   * Data management within and across data centers
   * Data-aware scheduling
   * Data-intensive computing applications
   * Eventual-consistency storage usage and management
* Programming models and tools
   * Map-reduce and its generalizations
   * Many-task computing middleware and applications
   * Parallel programming frameworks
   * Ensemble MPI techniques and frameworks
   * Service-oriented science applications
* Large-Scale Workflow Systems
   * Workflow system performance and scalability analysis
   * Scalability of workflow systems
   * Workflow infrastructure and e-Science middleware
   * Programming Paradigms and Models
* Large-Scale Many-Task Applications
   * High-throughput computing (HTC) applications
   * Data-intensive applications
   * Quasi-supercomputing applications, deployments, and experiences
   * Performance Evaluation
* Performance evaluation
   * Real systems
   * Simulations
   * Reliability of large systems


Paper Submission and Publication
---------------------------------------------------------------------------------------
Authors are invited to submit papers with unpublished, original work of not more than 6 pages of double column text using single spaced 10 point size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript guidelines; document templates can be found at http://www.acm.org/sigs/publications/proceedings-templates. The final 6 page papers in PDF format must be submitted online at https://cmt.research.microsoft.com/MTAGS2014/ before the deadline of August 25th, 2014 at 11:59PM PST (note that an abstract must be submitted 1 week prior to the deadline, on August 18th, 2014). Papers will be peer-reviewed, and accepted papers will be published in the workshop proceedings as part of the ACM digital library (in cooperation with SIGHPC). Notifications of the paper decisions will be sent out by September 22nd, 2014. Accepted workshop papers will be eligible for additional post-conference publication as journal articles in the IEEE Transaction on Cloud Computing, Special Issue on Many-Task Computing in the Cloud (papers will be due in February 2015, file:///C:/Users/iraicu/Documents/Webs/DataSys/events/TCC-MTC15/index.html). Submission implies the willingness of at least one of the authors to register and present the paper. For more information, please see http://datasys.cs.iit.edu/events/MTAGS14/.


Important Dates
---------------------------------------------------------------------------------------
*	Abstract Due:			August 18th, 2014
*	Papers Due:			August 25th, 2014
*	Notification of Acceptance:	September 22nd, 2014
*	Camera Ready Papers Due:	October  6th, 2014
*	Workshop Date:			November 16th, 2014



Committee Members
---------------------------------------------------------------------------------------
Workshop Chairs
*	Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory
*	Justin Wozniak, University of Chicago & Argonne National Laboratory
*	Ian Foster, University of Chicago & Argonne National Laboratory
*	Yong Zhao, University of Electronic Science and Technology of China

Steering Committee
*	David Abramson, Monash University, Australia
*	Jack Dongarra, University of Tennessee, USA
*	Geoffrey Fox, Indiana University, USA
*	Manish Parashar, Rutgers University, USA
*	Marc Snir, University of Illinois at Urbana Champaign, USA
*	Xian-He Sun, Illinois Institute of Technology, USA
*	Weimin Zheng, Tsinghua University, China

Technical Committee
*	Hasan Abbasi, Oak Ridge National Labs, USA
*	Tim Armstrong, University of Chicago, USA
*	Roger Barga, Microsoft, USA
*	Rajkumar Buyya University of Melbourne, Australia
*	Kyle Chard, University of Chicago, USA
*	Evangelinos Constantinos, Massachusetts Institute of Technology, USA
*	Catalin Dumitrescu, Fermi National Labs, USA
*	Haryadi Gunawi, University of Chicago, USA
*	Indranil Gupta, University of Illinois at Urbana Champaign, USA
*	Alexandru Iosup, Delft University of Technology, Netherlands
*	Florin Isaila, Universidad Carlos III de Madrid, Spain & Argonne National Laboratory, USA
*	Kamil Iskra, Argonne National Laboratory, USA
*	Daniel S. Katz, University of Chicago, USA
*	Jik-Soo Kim, Kristi, Korea
*	Scott A. Klasky, Oak Ridge National Labs, USA
*	Mike Lang, Los Alamos National Laboratory, USA
*	Tonglin Li, Illinois Institute of Technology, USA
*	Chris Moretti, Princeton University, USA
*	David O'Hallaron, Carnegie Mellon University, USA
*	Marlon Pierce, Indiana University, USA
*	Judy Qiu, Indiana University, USA
*	Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
*	Matei Ripeanu, University of British Columbia, Canada
*	Wei Tang, Argonne National Laboratory, USA
*	Edward Walker, Whitworth University, USA
*	Ke Wang, Illinois Institute of Technology, USA
*	Matthew Woitaszek, Walmart Labs, USA
*	Rich Wolski, University of California, Santa Barbara, USA
*	Zhifeng Yun, University of Houston, USA
*	Ziming Zheng, University of Chicago, USA


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================




From iraicu at cs.iit.edu  Sat Jul 12 09:20:36 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Sat, 12 Jul 2014 09:20:36 -0500
Subject: [Swift-devel] Call for Papers: IEEE Transactions on Cloud Computing
 - Special Issue on Many-Task Computing in the Cloud
Message-ID: <53C14434.3050806@cs.iit.edu>

Call for Papers

---------------------------------------------------------------------------------------
IEEE Transaction on Cloud Computing
Special Issue on Many-Task Computing in the Cloud
http://datasys.cs.iit.edu/events/TCC-MTC15/

=======================================================================================
The Special Issue on Many-Task Computing (MTC) in the Cloud will provide the scientific community a dedicated forum, within the prestigious IEEE Transactions on Cloud Computing journal, for presenting new research, development, and deployment efforts of loosely coupled large scale applications on Cloud Computing infrastructure. MTC, the theme of this special issue, encompasses loosely coupled applications, which are generally composed of many tasks to achieve some larger application goal. This special issue will cover challenges that can hamper efficiency and utilization in running applications on large-scale systems, such as local resource manager scalability and granularity, efficient utilization of raw hardware, parallel file-system contention and scalability, data management, I/O management, reliability at scale, and application scalability. We welcome paper submissions in theoretical, simulations, and systems topics with special consideration to papers addressing the intersection of petascale/exascale challenges with large-scale cloud computing. We seek submission of papers that present new, original and innovative ideas for the "first" time in TCC (Transactions on Cloud Computing). That means, submission of "extended versions" of already published works (e.g., conference/workshop papers) is not encouraged unless they contain significant number of "new and original" ideas/contributions along with more than 49% brand "new" material.

For more information on past workshops and special issues on Many-Task Computing, see http://datasys.cs.iit.edu/events/MTAGS/index.html. We ran a Special Issue on Many-Task Computing in the IEEE Transactions on Parallel and Distributed Systems (TPDS) which appeared in June 2011; the proceedings can be found online at http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm. The special issue editors also published a highly relevant paper that defines Many-Task Computing which was published in the inaugural MTAGS08 workshop, titled "Many-Task Computing for Grids and Supercomputers"; we encourage potential authors to read this paper, and to clearly articulate in your paper submissions how your papers are related to Many-Task Computing.

For more information on this special issue, please see http://datasys.cs.iit.edu/events/TCC-MTC15/.


Topics
---------------------------------------------------------------------------------------
We seek submission of papers that present new, original and innovative ideas for the "first" time in TCC (Transactions on Cloud Computing). That means, submission of "extended versions" of already published works (e.g., conference/workshop papers) will only be encouraged if they contain significant number of "new and original" ideas/contributions along with more than 49% brand "new" material. TCC expects submissions to be complete in all respects including author names, affiliation, bios etc. Manuscript should be 14 double column pages (all regular paper page limits include references and author biographies). We aim to cover topics related to Many-Task Computing and Cloud Computing. Topics of interest include:
* Compute Resource Management
   * Scheduling
   * Job execution frameworks
   * Local resource manager extensions
   * Performance evaluation of resource managers in use on large scale systems
   * Dynamic resource provisioning
   * Techniques to manage many-core resources and/or GPUs
   * Challenges and opportunities in running many-task workloads on Cloud Computing infrastructure
* Storage architectures and implementations
   * Distributed file systems
   * Parallel file systems
   * Distributed meta-data management
   * Content distribution systems for large data
   * Data caching frameworks and techniques
   * Data management within and across data centers
   * Data-aware scheduling
   * Data-intensive computing applications
   * Eventual-consistency storage usage and management
* Programming models and tools
   * Map-reduce and its generalizations
   * Many-task computing middleware and applications
   * Parallel programming frameworks
   * Ensemble MPI techniques and frameworks
   * Service-oriented science applications
* Large-Scale Workflow Systems
   * Workflow system performance and scalability analysis
   * Scalability of workflow systems
   * Workflow infrastructure and e-Science middleware
   * Programming Paradigms and Models
* Large-Scale Many-Task Applications
   * High-throughput computing (HTC) applications
   * Data-intensive applications
   * Quasi-supercomputing applications, deployments, and experiences
   * Performance Evaluation
* Performance evaluation
   * Real systems
   * Simulations
   * Reliability of large systems


Paper Submission and Publication
---------------------------------------------------------------------------------------

Authors are invited to submit unpublished and original work to the IEEE Transactions on Cloud Computing (TCC), Special Issue on Many-Task Computing in the Cloud. If the paper is extended from an initial work, the submission must contain at least 50% new material that can be qualified as ?brand? new ideas and results. The paper must be in the IEEE TCC format, namely 14 double-column pages or 30 single-column pages (Note: All regular paper page limits include references and author biographies). Please note that the double-column format will translate more readily into the final publication format. A double-column page is defined as a 7.875??10.75? page with 10-point type, 12-point vertical spacing, and 0.5 inch margins. A single-column page is defined as an 8.5??11? page with 12-point type and 24-point vertical spacing, containing approximately 250 words. All of the margins should be one inch (top, bottom, right and left). These length limits are taking into account reasonably-sized figures and references. Papers must be submitted using the submission system: https://mc.manuscriptcentral.com/tcc-cs, by selecting the special issue option ?SI-MTC?. For more information, please see http://datasys.cs.iit.edu/events/TCC-MTC15/.

Important Dates
---------------------------------------------------------------------------------------
*	Abstract Due:			February 2nd, 2015
*	Papers Due:			February 9th, 2015
* 	First round decisions:		May 18th, 2015
* 	Major Revisions if needed:	July 18th, 2015
* 	Final decisions:		August 18th, 2015
* 	Publication Date:		Fall 2015 (may vary depending on production queue)


Guest Editors
---------------------------------------------------------------------------------------
*	Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory
*	Justin Wozniak, University of Chicago & Argonne National Laboratory
*	Ian Foster, University of Chicago & Argonne National Laboratory
*	Yong Zhao, University of Electronic Science and Technology of China


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================




From wilde at anl.gov  Sun Jul 13 11:01:15 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 11:01:15 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
Message-ID: <53C2AD4B.9030605@anl.gov>

I did a quick scan of the new User Guide text that describes the 
proposed 0.96 config conventions.

I feel we need to re-examine the mechanism for backwards compatibility 
with 0.95, and decide on a few things.

- default property file names have changed: swift.properties vs swift.config

- the search path implemented by 0.95 is (I suspect) not yet carried 
forward here, and that needs discussion. My inclination is that it 
should be.

- the include mechanism is new.   I think its nice and likely is very 
useful, but I wonder how it will interact with or supplement the 
property search path.
Did that come from the underlying config classes used, or was that done 
from scratch? We should also consider hos this works with respect to the 
use of cpp in stc, if we see ourselves heading to the use of stc as a 
universal front end for the Swift toolchain.

- as we discussed before, but never resolved - we should decide if the 
documented/encouraged format will have colon separators and quoted 
string values. I prefer the visually simpler format without either, and 
I think 0.95 also allows an optional "=" separator as in standard 
properties files. ( http://en.wikipedia.org/wiki/.properties is a nice 
summary, and says " Several formats are possible for each line, 
including key=value, key = value, key:value, and key value.")

I think we can work all these issues out as we go.  I'm eager to test 
trunk in its current state before we address any of the issues above. 
Lets concentrate first on manual testing and on getting the property 
names stabilized and the more important semantic changes.

Also, we need to work out issues in application location specifications 
that were started but not yet fully developed in the 0.95 effort.  David 
and I discussed these, and we need to capture and post the 
recommendations that were still in progress for this.

- Mike

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140713/7822c921/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 13 13:01:19 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 13 Jul 2014 11:01:19 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C2AD4B.9030605@anl.gov>
References: <53C2AD4B.9030605@anl.gov>
Message-ID: <1405274479.16177.13.camel@echo>

On Sun, 2014-07-13 at 11:01 -0500, Michael Wilde wrote:
> - default property file names have changed: swift.properties vs swift.config

On purpose, to avoid left-overs from being confusing.

> 
> - the search path implemented by 0.95 is (I suspect) not yet carried 
> forward here, and that needs discussion. My inclination is that it 
> should be.
> 
> - the include mechanism is new.   I think its nice and likely is very 
> useful, but I wonder how it will interact with or supplement the 
> property search path.

We discussed this a few days ago. We had repeated issues with magically
loaded files from strange locations that the users took a long time to
find and fix. The solution that I saw in 0.95 was even more search
locations, which I think was not right.

So the philosophy in trunk is "either it stares at you or it isn't
there". And includes do this in a way that doesn't sacrifice
convenience.

> Did that come from the underlying config classes used, or was that done 
> from scratch? We should also consider hos this works with respect to the 
> use of cpp in stc, if we see ourselves heading to the use of stc as a 
> universal front end for the Swift toolchain.
> 
> - as we discussed before, but never resolved - we should decide if the 
> documented/encouraged format will have colon separators and quoted 
> string values.

You need to quote values that have special characters, such as "-" or
spaces, etc. Having to think about whether a particular string or not
does contain "special characters" to see if you are going to write
quotes or not is an effort that I do not think we should push the users
into. But it's a choice.

>  I prefer the visually simpler format without either, and 
> I think 0.95 also allows an optional "=" separator as in standard 
> properties files. ( http://en.wikipedia.org/wiki/.properties is a nice 
> summary, and says " Several formats are possible for each line, 
> including key=value, key = value, key:value, and key value.")

Give it some time :)
You might like it. I know it's not exactly the 0.95 config. It's not a
perfect world.

> 
> I think we can work all these issues out as we go.  I'm eager to test 
> trunk in its current state before we address any of the issues above. 
> Lets concentrate first on manual testing and on getting the property 
> names stabilized and the more important semantic changes.

Right.

> 
> Also, we need to work out issues in application location specifications 
> that were started but not yet fully developed in the 0.95 effort.  David 
> and I discussed these, and we need to capture and post the 
> recommendations that were still in progress for this.

Yes, please! I really wished I had seen a discussion on the 0.95 config
before it went into code.

Mihael



From wilde at anl.gov  Sun Jul 13 17:03:03 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 17:03:03 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405274479.16177.13.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
Message-ID: <53C30217.20501@anl.gov>


On 7/13/14, 1:01 PM, Mihael Hategan wrote:
>> - the include mechanism is new.   I think its nice and likely is very
>> >useful, but I wonder how it will interact with or supplement the
>> >property search path.
> We discussed this a few days ago. We had repeated issues with magically
> loaded files from strange locations that the users took a long time to
> find and fix. The solution that I saw in 0.95 was even more search
> locations, which I think was not right.
>
> So the philosophy in trunk is "either it stares at you or it isn't
> there". And includes do this in a way that doesn't sacrifice
> convenience.
>
I disagree with this. The properties search path in 0.95 was carefully 
thought out, and based on a lot of experience with users, even it that 
discussion did not take place on the list.

Unfortunately the trunk commits over-wrote the 0.95 documentation in the 
trunk userguide, but the search path was like this:

Location of swift.properties
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Swift searches for swift.properties files in multiple locations:

1. The etc/swift.properties file included with the Swift distribution.
2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates.
3. $HOME/.swift/swift.properties
4. swift.properties in your current directory.
5. Any property file you point to with the command line argument 
"-properties
<file>"

Settings get read in this order. Definitions in the later files will 
override
any previous definitions. For example, if you have execution.retries=10 in
$HOME/.swift/swift.properties, and execution.retries=0 in the 
swift.properties
in your current directory, execution.retries will be set to 0.


I thought this search path notion was logical, useful, and reflected the 
needs of users and site admins. It was a refinement of what existed in 
Swift going back many releases.

In 0.95 it was tied to the use of run directories (which I hope has not 
been removed from trunk!).

I propose that we re-institute this search path in trunk.

- Mike






From tim.g.armstrong at gmail.com  Sun Jul 13 17:05:07 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sun, 13 Jul 2014 17:05:07 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405274479.16177.13.camel@echo>
References: <53C2AD4B.9030605@anl.gov>
	<1405274479.16177.13.camel@echo>
Message-ID: <CAC0jiV60shBB7DAmBE83E+zFQvNOx76dpO+K=JAbip7rSNROtg@mail.gmail.com>

FYI I don't think we ever were set on having CPP as part of stc by default.
There are reasons not to (aside from the usual debates about misuse of
macros, it means that you're limited in what you can do syntactically, for
example multiline strings don't work properly).

It's still enabled by default but it's not actually needed.  I think we
actually made a preliminary decision to move away form it:
https://code.google.com/p/exm-issues/issues/detail?id=267.

- Tim


On Sun, Jul 13, 2014 at 1:01 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> On Sun, 2014-07-13 at 11:01 -0500, Michael Wilde wrote:
> > - default property file names have changed: swift.properties vs
> swift.config
>
> On purpose, to avoid left-overs from being confusing.
>
> >
> > - the search path implemented by 0.95 is (I suspect) not yet carried
> > forward here, and that needs discussion. My inclination is that it
> > should be.
> >
> > - the include mechanism is new.   I think its nice and likely is very
> > useful, but I wonder how it will interact with or supplement the
> > property search path.
>
> We discussed this a few days ago. We had repeated issues with magically
> loaded files from strange locations that the users took a long time to
> find and fix. The solution that I saw in 0.95 was even more search
> locations, which I think was not right.
>
> So the philosophy in trunk is "either it stares at you or it isn't
> there". And includes do this in a way that doesn't sacrifice
> convenience.
>
> > Did that come from the underlying config classes used, or was that done
> > from scratch? We should also consider hos this works with respect to the
> > use of cpp in stc, if we see ourselves heading to the use of stc as a
> > universal front end for the Swift toolchain.
> >
> > - as we discussed before, but never resolved - we should decide if the
> > documented/encouraged format will have colon separators and quoted
> > string values.
>
> You need to quote values that have special characters, such as "-" or
> spaces, etc. Having to think about whether a particular string or not
> does contain "special characters" to see if you are going to write
> quotes or not is an effort that I do not think we should push the users
> into. But it's a choice.
>
> >  I prefer the visually simpler format without either, and
> > I think 0.95 also allows an optional "=" separator as in standard
> > properties files. ( http://en.wikipedia.org/wiki/.properties is a nice
> > summary, and says " Several formats are possible for each line,
> > including key=value, key = value, key:value, and key value.")
>
> Give it some time :)
> You might like it. I know it's not exactly the 0.95 config. It's not a
> perfect world.
>
> >
> > I think we can work all these issues out as we go.  I'm eager to test
> > trunk in its current state before we address any of the issues above.
> > Lets concentrate first on manual testing and on getting the property
> > names stabilized and the more important semantic changes.
>
> Right.
>
> >
> > Also, we need to work out issues in application location specifications
> > that were started but not yet fully developed in the 0.95 effort.  David
> > and I discussed these, and we need to capture and post the
> > recommendations that were still in progress for this.
>
> Yes, please! I really wished I had seen a discussion on the 0.95 config
> before it went into code.
>
> Mihael
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140713/3b5e3c3d/attachment.html>

From delf.egge at hcu-hamburg.de  Sun Jul 13 17:22:47 2014
From: delf.egge at hcu-hamburg.de (Delf Egge HCU)
Date: Mon, 14 Jul 2014 00:22:47 +0200
Subject: [Swift-devel] Question: Reading from keyboard
In-Reply-To: <mailman.32075.1405289104.3916.swift-devel@ci.uchicago.edu>
References: <mailman.32075.1405289104.3916.swift-devel@ci.uchicago.edu>
Message-ID: <0AA7F6B2-1DC9-4657-A451-EFE7940424ED@hcu-hamburg.de>

Dear people! 

Could you please help me and let me know 

how to input characters (strings) from the keyboard with Swift? 

I am only aware of the C language "scanf". 

Thank you very much in advance. 

Kind regards, 
Delf Egge 

delf.egge at hcu-hamburg.de

Sent from my iPhone. 

> Am 14.07.2014 um 00:05 schrieb swift-devel-request at ci.uchicago.edu:
> 
> Send Swift-devel mailing list 

Rest deleted. 


From wilde at anl.gov  Sun Jul 13 17:40:38 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 17:40:38 -0500
Subject: [Swift-devel] Question: Reading from keyboard
In-Reply-To: <0AA7F6B2-1DC9-4657-A451-EFE7940424ED@hcu-hamburg.de>
References: <mailman.32075.1405289104.3916.swift-devel@ci.uchicago.edu>
	<0AA7F6B2-1DC9-4657-A451-EFE7940424ED@hcu-hamburg.de>
Message-ID: <53C30AE6.7020104@anl.gov>

Delf,

If you are asking about Apple's Swift language, please refer to 
https://developer.apple.com/swift/

If you are asking about the Swift parallel scripting language 
(http://swift-lang.org), you should join the swift-user email list and 
ask this question there.

Having said that, in our Swift language there is seldom a reason to read 
from stdin, although one could envision such cases.

You can read from stdin in a limited manner like this:

$ cat stdin.swift
type file;

string s = readData("/dev/stdin");
trace(s);

$ swift stdin.swift
Swift 0.94.1 swift-r7114 cog-r3803

RunID: 20140713-2234-4r8sfd3b
Progress:  time: Sun, 13 Jul 2014 22:34:59 +0000
This is my input on stdin.
^D
SwiftScript trace: This is my input on stdin.
Final status: Sun, 13 Jul 2014 22:35:11 +0000
$

All of stdin until an EOF (^D) is read by readData( ).
There is no line separation if you read into a scalar.

If you read into a string array, each line of stdin is assigned to an 
element of the array.

One could envision a read function that reads stdin a line at a time, to 
enable an interactive Swift script, but to date this need has not arisen.

- Mike

On 7/13/14, 5:22 PM, Delf Egge HCU wrote:
> Dear people!
>
> Could you please help me and let me know
>
> how to input characters (strings) from the keyboard with Swift?
>
> I am only aware of the C language "scanf".
>
> Thank you very much in advance.
>
> Kind regards,
> Delf Egge
>
> delf.egge at hcu-hamburg.de
>
> Sent from my iPhone.
>
>> Am 14.07.2014 um 00:05 schrieb swift-devel-request at ci.uchicago.edu:
>>
>> Send Swift-devel mailing list
> Rest deleted.
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Sun Jul 13 18:37:18 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 13 Jul 2014 16:37:18 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C30217.20501@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov>
Message-ID: <1405294638.17861.33.camel@echo>

I know the feeling. I get it too. I see some new stuff and I was getting
used to the old stuff, and I get this immediate anger about the new
stuff and the feeling that it's stupid and I want the old stuff back.

All I'm saying is give it some time. Try it out a bit, see how you would
do something with it, etc. It's not set in stone. We have SVN and we can
always go back.

Ultimately the questions are: can I do what I was doing before, and how
much effort does that require. Also what are the new things that I can
do and could not do before, and how much better/worse is the overall
process. I know what the previous thing could do, and you can still do
that, just not in the exact same way.

Some more inline...

On Sun, 2014-07-13 at 17:03 -0500, Michael Wilde wrote:
> On 7/13/14, 1:01 PM, Mihael Hategan wrote:
> >> - the include mechanism is new.   I think its nice and likely is very
> >> >useful, but I wonder how it will interact with or supplement the
> >> >property search path.
> > We discussed this a few days ago. We had repeated issues with magically
> > loaded files from strange locations that the users took a long time to
> > find and fix. The solution that I saw in 0.95 was even more search
> > locations, which I think was not right.
> >
> > So the philosophy in trunk is "either it stares at you or it isn't
> > there". And includes do this in a way that doesn't sacrifice
> > convenience.
> >
> I disagree with this. The properties search path in 0.95 was carefully 
> thought out, and based on a lot of experience with users, even it that 
> discussion did not take place on the list.

You can write a config file that includes all the different config files
and put it in swift/etc, if, as a user, that is what you think is best.

> 
> Unfortunately the trunk commits over-wrote the 0.95 documentation in the 
> trunk userguide, but the search path was like this:
> 
> Location of swift.properties
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Swift searches for swift.properties files in multiple locations:
> 
> 1. The etc/swift.properties file included with the Swift distribution.
> 2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates.
> 3. $HOME/.swift/swift.properties
> 4. swift.properties in your current directory.
> 5. Any property file you point to with the command line argument 
> "-properties
> <file>"
> 
> Settings get read in this order. Definitions in the later files will 
> override
> any previous definitions. For example, if you have execution.retries=10 in
> $HOME/.swift/swift.properties, and execution.retries=0 in the 
> swift.properties
> in your current directory, execution.retries will be set to 0.

Yes. Well, I think this is bad. Imagine a user having to mentally go
through this sequence in order to figure out what's being set and where.
It's actually nearly impossible mentally. You have to go through the
files and remember the order.

The biggest problems we've had so far in terms of the configuration we
had were with this. A file in the search path magically loaded without
the user's knowledge. Even recently you and Yadu have experienced this,
and were wondering where some configuration value comes from that was
causing a run to break. And it took you a few hours and I had to get
involved to sort it out. And It was all documented. See
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281

So no, I do not think that this is a good idea.

> 
> I thought this search path notion was logical, useful, and reflected the 
> needs of users and site admins. It was a refinement of what existed in 
> Swift going back many releases.

It added extra steps to something that already had too many steps.

That said, the new stuff doesn't say you can't do that. Just that you
have to actually say it yourself that you want to do that. The reason we
had multiple search paths is because we did not support includes, so we
needed a mechanism to override only a few things. But you can simulate
it with includes and customize it in whatever way you want and it would
be obvious what's coming from where.

The point is to give the user a simple and powerful tool to do what they
need.

> 
> In 0.95 it was tied to the use of run directories (which I hope has not 
> been removed from trunk!).

No. It's still there. I like it. I only renamed runxxx.log to swift.log,
since I did not se why everything had a name that did not include the
run id except for the log. We can change this back if there is a reason.

We do probably need to fix a few issues with it. It will break with
concurrent starts of swift, and it will break on filesystems that don't
list directories in some particular order.




From tim.g.armstrong at gmail.com  Sun Jul 13 20:04:30 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Sun, 13 Jul 2014 20:04:30 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405294638.17861.33.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
Message-ID: <CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>

My two cents: the five step search path seems logical and I can come up
with scenarios where someone would want to have a particular setting at a
particular level in the hierarchy.

It reminds me though of some of the OOP debates about inheritance, where
you have carefully thought out, deep, inheritance hierarchies, carefully
engineered to reuse code wherever possible.  The problem tends to be that
as a developer reading a bit of code, you've got no idea which overridden
method is being called, except maybe with the help of an IDE.  There's not
really a right answer, just that sometimes the smart complicated way is the
best, and sometimes the dumb but simple way is the best.

Maybe if having 4/5 levels of configuration is "the right thing" that users
should do, all of the sample and tutorial properties files that users will
copy could just explicitly include their predecessor?  I.e. use the
explicit mechanism but encourage the desired approach by convention.

With the more complex inheritance mechanism, it seems like it is almost
necessary to have a tool that shows all of the currently applied settings
and where they come from.

- Tim


On Sun, Jul 13, 2014 at 6:37 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> I know the feeling. I get it too. I see some new stuff and I was getting
> used to the old stuff, and I get this immediate anger about the new
> stuff and the feeling that it's stupid and I want the old stuff back.
>
> All I'm saying is give it some time. Try it out a bit, see how you would
> do something with it, etc. It's not set in stone. We have SVN and we can
> always go back.
>
> Ultimately the questions are: can I do what I was doing before, and how
> much effort does that require. Also what are the new things that I can
> do and could not do before, and how much better/worse is the overall
> process. I know what the previous thing could do, and you can still do
> that, just not in the exact same way.
>
> Some more inline...
>
> On Sun, 2014-07-13 at 17:03 -0500, Michael Wilde wrote:
> > On 7/13/14, 1:01 PM, Mihael Hategan wrote:
> > >> - the include mechanism is new.   I think its nice and likely is very
> > >> >useful, but I wonder how it will interact with or supplement the
> > >> >property search path.
> > > We discussed this a few days ago. We had repeated issues with magically
> > > loaded files from strange locations that the users took a long time to
> > > find and fix. The solution that I saw in 0.95 was even more search
> > > locations, which I think was not right.
> > >
> > > So the philosophy in trunk is "either it stares at you or it isn't
> > > there". And includes do this in a way that doesn't sacrifice
> > > convenience.
> > >
> > I disagree with this. The properties search path in 0.95 was carefully
> > thought out, and based on a lot of experience with users, even it that
> > discussion did not take place on the list.
>
> You can write a config file that includes all the different config files
> and put it in swift/etc, if, as a user, that is what you think is best.
>
> >
> > Unfortunately the trunk commits over-wrote the 0.95 documentation in the
> > trunk userguide, but the search path was like this:
> >
> > Location of swift.properties
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Swift searches for swift.properties files in multiple locations:
> >
> > 1. The etc/swift.properties file included with the Swift distribution.
> > 2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates.
> > 3. $HOME/.swift/swift.properties
> > 4. swift.properties in your current directory.
> > 5. Any property file you point to with the command line argument
> > "-properties
> > <file>"
> >
> > Settings get read in this order. Definitions in the later files will
> > override
> > any previous definitions. For example, if you have execution.retries=10
> in
> > $HOME/.swift/swift.properties, and execution.retries=0 in the
> > swift.properties
> > in your current directory, execution.retries will be set to 0.
>
> Yes. Well, I think this is bad. Imagine a user having to mentally go
> through this sequence in order to figure out what's being set and where.
> It's actually nearly impossible mentally. You have to go through the
> files and remember the order.
>
> The biggest problems we've had so far in terms of the configuration we
> had were with this. A file in the search path magically loaded without
> the user's knowledge. Even recently you and Yadu have experienced this,
> and were wondering where some configuration value comes from that was
> causing a run to break. And it took you a few hours and I had to get
> involved to sort it out. And It was all documented. See
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281
>
> So no, I do not think that this is a good idea.
>
> >
> > I thought this search path notion was logical, useful, and reflected the
> > needs of users and site admins. It was a refinement of what existed in
> > Swift going back many releases.
>
> It added extra steps to something that already had too many steps.
>
> That said, the new stuff doesn't say you can't do that. Just that you
> have to actually say it yourself that you want to do that. The reason we
> had multiple search paths is because we did not support includes, so we
> needed a mechanism to override only a few things. But you can simulate
> it with includes and customize it in whatever way you want and it would
> be obvious what's coming from where.
>
> The point is to give the user a simple and powerful tool to do what they
> need.
>
> >
> > In 0.95 it was tied to the use of run directories (which I hope has not
> > been removed from trunk!).
>
> No. It's still there. I like it. I only renamed runxxx.log to swift.log,
> since I did not se why everything had a name that did not include the
> run id except for the log. We can change this back if there is a reason.
>
> We do probably need to fix a few issues with it. It will break with
> concurrent starts of swift, and it will break on filesystems that don't
> list directories in some particular order.
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140713/1d773719/attachment.html>

From wilde at anl.gov  Sun Jul 13 20:17:53 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 20:17:53 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
References: <53C2AD4B.9030605@anl.gov>	<1405274479.16177.13.camel@echo>	<53C30217.20501@anl.gov>	<1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
Message-ID: <53C32FC1.70507@anl.gov>

I agree about the logical nature of the search path.  But it was 
designed so that most users would see a very simple view of 
configuration. Its not really like a deep inheritance tree.

The concept was this:

 > 1. The etc/swift.properties file included with the Swift distribution.

These would be basic things like localhost, and maybe a few simple sites 
for generic cloud pools and sets of ad-hoc nodes.
Things that should enable you to just download Swift and try a few 
things without thinking (or even knowing) about sites.

 > 2. $SWIFT_SITE_CONF/swift.properties - used for defining site templates.

This was for site admins to make, e.g., Swift modules, so that when you 
say "module load swift" you get a set of reasonable site definitions for 
local clusters and queues.

 > 3. $HOME/.swift/swift.properties

This would allow the user to set some preferences, like whether they 
wanted Swift retry to be on or off by default.

 > 4. swift.properties in your current directory.

This was seen as the *main* property file that a user would create, 
alongside their .swift file, for a given scripting application.  Its 
where anything specific to running a given script would be set.

 > 5. Any property file you point to with the command line argument

This was seen as a way to specify a remote configuration file, and was 
expected to be seldom used by most users.

Regarding the tool, I agree totally, and there was indeed such a thing 
provided in 0.95:

To verify what files are being read, and what values will be set, run:
-----
$ swift -listconfig
-----

Ive not checked its output, but both it, and the swift log, should list:

a)  a set of all the properties files that were read in the run (or in 
the current environment)

b) a hierarchical listing of all properties (including site attributes), 
listed one attribute per line.
For each attribute, the output can append a simple [n] notation after 
each attribute stating which of the N property files that attribute was 
set from.

- Mike


On 7/13/14, 8:04 PM, Tim Armstrong wrote:
> My two cents: the five step search path seems logical and I can come 
> up with scenarios where someone would want to have a particular 
> setting at a particular level in the hierarchy.
>
> It reminds me though of some of the OOP debates about inheritance, 
> where you have carefully thought out, deep, inheritance hierarchies, 
> carefully engineered to reuse code wherever possible.  The problem 
> tends to be that as a developer reading a bit of code, you've got no 
> idea which overridden method is being called, except maybe with the 
> help of an IDE.  There's not really a right answer, just that 
> sometimes the smart complicated way is the best, and sometimes the 
> dumb but simple way is the best.
>
> Maybe if having 4/5 levels of configuration is "the right thing" that 
> users should do, all of the sample and tutorial properties files that 
> users will copy could just explicitly include their predecessor?  I.e. 
> use the explicit mechanism but encourage the desired approach by 
> convention.
>
> With the more complex inheritance mechanism, it seems like it is 
> almost necessary to have a tool that shows all of the currently 
> applied settings and where they come from.
>
> - Tim
>
>
> On Sun, Jul 13, 2014 at 6:37 PM, Mihael Hategan <hategan at mcs.anl.gov 
> <mailto:hategan at mcs.anl.gov>> wrote:
>
>     I know the feeling. I get it too. I see some new stuff and I was
>     getting
>     used to the old stuff, and I get this immediate anger about the new
>     stuff and the feeling that it's stupid and I want the old stuff back.
>
>     All I'm saying is give it some time. Try it out a bit, see how you
>     would
>     do something with it, etc. It's not set in stone. We have SVN and
>     we can
>     always go back.
>
>     Ultimately the questions are: can I do what I was doing before,
>     and how
>     much effort does that require. Also what are the new things that I can
>     do and could not do before, and how much better/worse is the overall
>     process. I know what the previous thing could do, and you can still do
>     that, just not in the exact same way.
>
>     Some more inline...
>
>     On Sun, 2014-07-13 at 17:03 -0500, Michael Wilde wrote:
>     > On 7/13/14, 1:01 PM, Mihael Hategan wrote:
>     > >> - the include mechanism is new.   I think its nice and likely
>     is very
>     > >> >useful, but I wonder how it will interact with or supplement the
>     > >> >property search path.
>     > > We discussed this a few days ago. We had repeated issues with
>     magically
>     > > loaded files from strange locations that the users took a long
>     time to
>     > > find and fix. The solution that I saw in 0.95 was even more search
>     > > locations, which I think was not right.
>     > >
>     > > So the philosophy in trunk is "either it stares at you or it isn't
>     > > there". And includes do this in a way that doesn't sacrifice
>     > > convenience.
>     > >
>     > I disagree with this. The properties search path in 0.95 was
>     carefully
>     > thought out, and based on a lot of experience with users, even
>     it that
>     > discussion did not take place on the list.
>
>     You can write a config file that includes all the different config
>     files
>     and put it in swift/etc, if, as a user, that is what you think is
>     best.
>
>     >
>     > Unfortunately the trunk commits over-wrote the 0.95
>     documentation in the
>     > trunk userguide, but the search path was like this:
>     >
>     > Location of swift.properties
>     > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     > Swift searches for swift.properties files in multiple locations:
>     >
>     > 1. The etc/swift.properties file included with the Swift
>     distribution.
>     > 2. $SWIFT_SITE_CONF/swift.properties - used for defining site
>     templates.
>     > 3. $HOME/.swift/swift.properties
>     > 4. swift.properties in your current directory.
>     > 5. Any property file you point to with the command line argument
>     > "-properties
>     > <file>"
>     >
>     > Settings get read in this order. Definitions in the later files will
>     > override
>     > any previous definitions. For example, if you have
>     execution.retries=10 in
>     > $HOME/.swift/swift.properties, and execution.retries=0 in the
>     > swift.properties
>     > in your current directory, execution.retries will be set to 0.
>
>     Yes. Well, I think this is bad. Imagine a user having to mentally go
>     through this sequence in order to figure out what's being set and
>     where.
>     It's actually nearly impossible mentally. You have to go through the
>     files and remember the order.
>
>     The biggest problems we've had so far in terms of the configuration we
>     had were with this. A file in the search path magically loaded without
>     the user's knowledge. Even recently you and Yadu have experienced
>     this,
>     and were wondering where some configuration value comes from that was
>     causing a run to break. And it took you a few hours and I had to get
>     involved to sort it out. And It was all documented. See
>     https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1281
>
>     So no, I do not think that this is a good idea.
>
>     >
>     > I thought this search path notion was logical, useful, and
>     reflected the
>     > needs of users and site admins. It was a refinement of what
>     existed in
>     > Swift going back many releases.
>
>     It added extra steps to something that already had too many steps.
>
>     That said, the new stuff doesn't say you can't do that. Just that you
>     have to actually say it yourself that you want to do that. The
>     reason we
>     had multiple search paths is because we did not support includes,
>     so we
>     needed a mechanism to override only a few things. But you can simulate
>     it with includes and customize it in whatever way you want and it
>     would
>     be obvious what's coming from where.
>
>     The point is to give the user a simple and powerful tool to do
>     what they
>     need.
>
>     >
>     > In 0.95 it was tied to the use of run directories (which I hope
>     has not
>     > been removed from trunk!).
>
>     No. It's still there. I like it. I only renamed runxxx.log to
>     swift.log,
>     since I did not se why everything had a name that did not include the
>     run id except for the log. We can change this back if there is a
>     reason.
>
>     We do probably need to fix a few issues with it. It will break with
>     concurrent starts of swift, and it will break on filesystems that
>     don't
>     list directories in some particular order.
>
>
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140713/b5f4767e/attachment.html>

From hategan at mcs.anl.gov  Sun Jul 13 20:55:15 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 13 Jul 2014 18:55:15 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
Message-ID: <1405302915.18573.37.camel@echo>

Inheritance is nice and we want that. It's there, not going away. But it
is being made explicit rather than implicit. In an OO language it's the
same as saying that if you inherit from a class, you need to tell the
compiler what the class is instead of only allowing classes whose names
start with B to inherit from classes whose names start with GRZ who in
turn can only inherit from a class called BLA, and you don't get to say
which; the compiler automatically picks them for you.

The five step path is logical and I too can come up with scenarios where
it would be needed. I can also come up with scenarios where something
else is logical, and there are examples of scenarios in which a
similarly stepped path is a problem. 

So I think we are arguing very different things here. What I do not like
is lenient defaults. Like $PWD/swift.conf. If the user creates a file
siwft.conf, Swift will happily run. The user will probably only notice
it after a lot of work and pain. Similar story with
~/.swift/swift.properties. But I have never heard one person complain
that they have to say -sites.file on the command line. 

On Sun, 2014-07-13 at 20:04 -0500, Tim Armstrong wrote:
> My two cents: the five step search path seems logical and I can come up
> with scenarios where someone would want to have a particular setting at a
> particular level in the hierarchy.
> 
> It reminds me though of some of the OOP debates about inheritance, where
> you have carefully thought out, deep, inheritance hierarchies, carefully
> engineered to reuse code wherever possible.  The problem tends to be that
> as a developer reading a bit of code, you've got no idea which overridden
> method is being called, except maybe with the help of an IDE.  There's not
> really a right answer, just that sometimes the smart complicated way is the
> best, and sometimes the dumb but simple way is the best.
> 
> Maybe if having 4/5 levels of configuration is "the right thing" that users
> should do, all of the sample and tutorial properties files that users will
> copy could just explicitly include their predecessor?  I.e. use the
> explicit mechanism but encourage the desired approach by convention.

The "desired approach" here is our interpretation of the feedback we got
from the users. I don't think we should encourage them to do what we
think they said they want to do. They probably know better.

> 
> With the more complex inheritance mechanism, it seems like it is almost
> necessary to have a tool that shows all of the currently applied settings
> and where they come from.

I agree. This is currently in the log, but it can be better.

Mihael



From wilde at anl.gov  Sun Jul 13 20:59:28 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 20:59:28 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405302915.18573.37.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>	
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>	
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo>
Message-ID: <53C33980.1070608@anl.gov>


On 7/13/14, 8:55 PM, Mihael Hategan wrote:
> But I have never heard one person complain
> that they have to say -sites.file on the command line.
The fact that the standard swift invocation line today is:

   swift -tc.file tc.data -sites.file sites.xml -config swift.properties

is absolutely insane, and its what 0.95 nicely fixed.

Users don't complain - they go find simpler languages.

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Sun Jul 13 21:25:45 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 13 Jul 2014 19:25:45 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C33980.1070608@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
Message-ID: <1405304745.18573.63.camel@echo>

On Sun, 2014-07-13 at 20:59 -0500, Michael Wilde wrote:
> On 7/13/14, 8:55 PM, Mihael Hategan wrote:
> > But I have never heard one person complain
> > that they have to say -sites.file on the command line.
> The fact that the standard swift invocation line today is:
> 
>    swift -tc.file tc.data -sites.file sites.xml -config swift.properties

Most users would have a run script for this. It's trivial to write, and
hard to make unnoticeable mistakes with it.

> 
> is absolutely insane, and its what 0.95 nicely fixed.

Not insane. Just what you should expect from a command-line tool. Like,
gcc -Wno-write-strings -L/usr/local/lib, etc. 

Having 5 fallback configuration file locations is more uncommon I think.

> 
> Users don't complain - they go find simpler languages.
> 

Maybe. My impression was that it was hard to understand what was
happening behind their backs, which I thought was a somewhat known
problem in design for which the commonly accepted solutions were: have a
single, well specified way to do things, and don't have magical
behavior. 

I'm getting the feeling that you would rather have everything reverted
to what was in 0.95. Let me know if this is the case. It should be
pretty straightforward to do.

Mihael



From wilde at anl.gov  Sun Jul 13 21:27:21 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sun, 13 Jul 2014 21:27:21 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405304745.18573.63.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>		
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>		
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>	
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo>
Message-ID: <53C34009.9010302@anl.gov>


On 7/13/14, 9:25 PM, Mihael Hategan wrote:
> I'm getting the feeling that you would rather have everything reverted
> to what was in 0.95. Let me know if this is the case. It should be
> pretty straightforward to do.
I want to keep the new config mechanism you just added, and bring back 
the search paths as they were in 0.95.

- Mike


From hategan at mcs.anl.gov  Sun Jul 13 22:02:37 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sun, 13 Jul 2014 20:02:37 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C34009.9010302@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
Message-ID: <1405306957.18573.85.camel@echo>

Sorry. This discussion went a bit off. I apologize.

Search path. There are some changes. Here are the details:

1. swift/etc/swift.conf is still there
2. SWIFT_SITE_CONF is still there. It makes good sense.
3. ~/.swift/swift.conf is gone. SWIFT_SITE_CONF does the same and is
explicit. Can be added back.
4. ./swift.conf is gone for reasons of magicality. Can also be added
back.
5. -config on the command line is still there.

The fundamental difference is that there is no "if it's not there,
ignore silently and continue" behavior and there is no automated
chaining. The user has to say what they want. It's a one-time operation
per file. Very likely less work than the actual contents of the file,
but less error-prone. It's a choice. That of safety vs. convenience. We
can bias either way and supplement with tools.

We could and should discuss the merits of what has changed. It might
help if we do it one issue at a time.

Mihael



From hategan at mcs.anl.gov  Mon Jul 14 13:00:28 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 14 Jul 2014 11:00:28 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405306957.18573.85.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo>
Message-ID: <1405360828.24817.6.camel@echo>

Hi,

So, first, when I say "gone" that means "it's not there right now in
trunk". This is work in progress, and I was hoping to get feedback from
everybody, hopefully based on use cases and experiences. This change
brings an opportunity to re-visit some of the things we have and see if
they make sense in the new context and make the right long-term
decisions.

The first question is whether ~/.swift/swift.conf and ./swift.conf
should be explicitly in the search path, since SWIFT_SITE_CONF can do
both:

export SWIFT_SITE_CONF=~/.swift/swift.conf
export SWIFT_SITE_CONF=./swift.conf

Mihael

On Sun, 2014-07-13 at 20:02 -0700, Mihael Hategan wrote:
> Sorry. This discussion went a bit off. I apologize.
> 
> Search path. There are some changes. Here are the details:
> 
> 1. swift/etc/swift.conf is still there
> 2. SWIFT_SITE_CONF is still there. It makes good sense.
> 3. ~/.swift/swift.conf is gone. SWIFT_SITE_CONF does the same and is
> explicit. Can be added back.
> 4. ./swift.conf is gone for reasons of magicality. Can also be added
> back.
> 5. -config on the command line is still there.
> 
> The fundamental difference is that there is no "if it's not there,
> ignore silently and continue" behavior and there is no automated
> chaining. The user has to say what they want. It's a one-time operation
> per file. Very likely less work than the actual contents of the file,
> but less error-prone. It's a choice. That of safety vs. convenience. We
> can bias either way and supplement with tools.
> 
> We could and should discuss the merits of what has changed. It might
> help if we do it one issue at a time.
> 
> Mihael
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From yadunand at uchicago.edu  Mon Jul 14 14:36:51 2014
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Mon, 14 Jul 2014 14:36:51 -0500
Subject: [Swift-devel] Swift on AWS-EC2 [Request for comments]
Message-ID: <53C43153.6090704@uchicago.edu>

Hi Everyone,

I've added support for Amazon's-EC2 to the swift-on-cloud repository and 
tutorial. Please try the setup and documentation.

If you are on the swift allocation on AWS, you'd only need to get the 
IAM credentials file from the swift aws console:
https://swift.signin.aws.amazon.com/console 
<https://www.google.com/url?q=https%3A%2F%2Fswift.signin.aws.amazon.com%2Fconsole&sa=D&sntz=1&usg=AFQjCNGCcWeMITvu7cmbPZWMClr-pk96VA>
Otherwise, you'd have to setup billing, and with micro instances prices 
are now the same as Google's Compute Engine (around $0.013/Hr/micro 
instance)

Cloud setup online doc:
https://github.com/yadudoc/swift-on-cloud/tree/master/aws

Git repo to clone:
https://github.com/yadudoc/swift-on-cloud.git

Once setup is done, do "connect headnode" and you can run the 
swift-cloud-tutorial
under your $HOME/swift-on-cloud/swift-cloud-tutorial folder on the 
headnode instance.

Feedback would be much appreciated.

PS: Remember to shut-down instances using the "dissolve" command.


Thanks!
Yadu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140714/ca033ecf/attachment.html>

From hategan at mcs.anl.gov  Mon Jul 14 15:06:38 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 14 Jul 2014 13:06:38 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405360828.24817.6.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
Message-ID: <1405368398.25594.7.camel@echo>

If there are no comments or suggestions besides what has already been
said, I'll keep the 5 step search path exactly as it is in 0.95. Please
do comment if you have something to say. We're less likely to re-visit
the issue in the future.

Anyway, the second issue is whether we want automatic or explicit
inheritance. In other words, do we want all of the 5 steps to be loaded
in order, non-existing files being silently skipped, or do we want the
first file in the search path to have to explicitly include others. Or
do we want a mixed behavior (such as SWIFT_SITE_CONF and ./swift.conf
both loaded automaticaly but not others)?

Mihael

On Mon, 2014-07-14 at 11:00 -0700, Mihael Hategan wrote:
> Hi,
> 
> So, first, when I say "gone" that means "it's not there right now in
> trunk". This is work in progress, and I was hoping to get feedback from
> everybody, hopefully based on use cases and experiences. This change
> brings an opportunity to re-visit some of the things we have and see if
> they make sense in the new context and make the right long-term
> decisions.
> 
> The first question is whether ~/.swift/swift.conf and ./swift.conf
> should be explicitly in the search path, since SWIFT_SITE_CONF can do
> both:
> 
> export SWIFT_SITE_CONF=~/.swift/swift.conf
> export SWIFT_SITE_CONF=./swift.conf
> 
> Mihael
> 
> On Sun, 2014-07-13 at 20:02 -0700, Mihael Hategan wrote:
> > Sorry. This discussion went a bit off. I apologize.
> > 
> > Search path. There are some changes. Here are the details:
> > 
> > 1. swift/etc/swift.conf is still there
> > 2. SWIFT_SITE_CONF is still there. It makes good sense.
> > 3. ~/.swift/swift.conf is gone. SWIFT_SITE_CONF does the same and is
> > explicit. Can be added back.
> > 4. ./swift.conf is gone for reasons of magicality. Can also be added
> > back.
> > 5. -config on the command line is still there.
> > 
> > The fundamental difference is that there is no "if it's not there,
> > ignore silently and continue" behavior and there is no automated
> > chaining. The user has to say what they want. It's a one-time operation
> > per file. Very likely less work than the actual contents of the file,
> > but less error-prone. It's a choice. That of safety vs. convenience. We
> > can bias either way and supplement with tools.
> > 
> > We could and should discuss the merits of what has changed. It might
> > help if we do it one issue at a time.
> > 
> > Mihael
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From tim.g.armstrong at gmail.com  Mon Jul 14 15:57:12 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Mon, 14 Jul 2014 15:57:12 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405368398.25594.7.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
Message-ID: <CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>

I think the implicit versus explicit thing is a matter of taste; that said,
as a personal preference it seems more elegant and convenient to have a
powerful but explicit include mechanism so that there's an easy and obvious
way to include alternative files at any stage in the process.

A few observations/questions:
* As a user I'd find it really surprising that the implicit ./swift.conf
still is in effect when I provided an explicit command line config.  I'm
not aware of any other systems that behave like this.
* If you invoke a Swift script from another directory, does the config file
in your current directory or the script directory take effect?  It seems
counterintuitive to me that you would have to cd into the script directory
when you're already explicitly giving the path to the Swift script.
* Is there a way I can temporarily disable a config file on the search
path, beyond moving the file or overriding all of the keys?  If so, is this
method discoverable to users?
* If ./swift.conf is meant to correspond to a particular Swift script,
would it make more sense to have the config name match the script name?

- Tim


On Mon, Jul 14, 2014 at 3:06 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> If there are no comments or suggestions besides what has already been
> said, I'll keep the 5 step search path exactly as it is in 0.95. Please
> do comment if you have something to say. We're less likely to re-visit
> the issue in the future.
>
> Anyway, the second issue is whether we want automatic or explicit
> inheritance. In other words, do we want all of the 5 steps to be loaded
> in order, non-existing files being silently skipped, or do we want the
> first file in the search path to have to explicitly include others. Or
> do we want a mixed behavior (such as SWIFT_SITE_CONF and ./swift.conf
> both loaded automaticaly but not others)?
>
> Mihael
>
> On Mon, 2014-07-14 at 11:00 -0700, Mihael Hategan wrote:
> > Hi,
> >
> > So, first, when I say "gone" that means "it's not there right now in
> > trunk". This is work in progress, and I was hoping to get feedback from
> > everybody, hopefully based on use cases and experiences. This change
> > brings an opportunity to re-visit some of the things we have and see if
> > they make sense in the new context and make the right long-term
> > decisions.
> >
> > The first question is whether ~/.swift/swift.conf and ./swift.conf
> > should be explicitly in the search path, since SWIFT_SITE_CONF can do
> > both:
> >
> > export SWIFT_SITE_CONF=~/.swift/swift.conf
> > export SWIFT_SITE_CONF=./swift.conf
> >
> > Mihael
> >
> > On Sun, 2014-07-13 at 20:02 -0700, Mihael Hategan wrote:
> > > Sorry. This discussion went a bit off. I apologize.
> > >
> > > Search path. There are some changes. Here are the details:
> > >
> > > 1. swift/etc/swift.conf is still there
> > > 2. SWIFT_SITE_CONF is still there. It makes good sense.
> > > 3. ~/.swift/swift.conf is gone. SWIFT_SITE_CONF does the same and is
> > > explicit. Can be added back.
> > > 4. ./swift.conf is gone for reasons of magicality. Can also be added
> > > back.
> > > 5. -config on the command line is still there.
> > >
> > > The fundamental difference is that there is no "if it's not there,
> > > ignore silently and continue" behavior and there is no automated
> > > chaining. The user has to say what they want. It's a one-time operation
> > > per file. Very likely less work than the actual contents of the file,
> > > but less error-prone. It's a choice. That of safety vs. convenience. We
> > > can bias either way and supplement with tools.
> > >
> > > We could and should discuss the merits of what has changed. It might
> > > help if we do it one issue at a time.
> > >
> > > Mihael
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140714/013ffbde/attachment.html>

From hategan at mcs.anl.gov  Mon Jul 14 16:29:32 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 14 Jul 2014 14:29:32 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
Message-ID: <1405373372.26058.30.camel@echo>

Thanks. Inline...

On Mon, 2014-07-14 at 15:57 -0500, Tim Armstrong wrote:
> I think the implicit versus explicit thing is a matter of taste; that said,
> as a personal preference it seems more elegant and convenient to have a
> powerful but explicit include mechanism so that there's an easy and obvious
> way to include alternative files at any stage in the process.
> 
> A few observations/questions:
> * As a user I'd find it really surprising that the implicit ./swift.conf
> still is in effect when I provided an explicit command line config.  I'm
> not aware of any other systems that behave like this.

That seems to be what happens in 0.95. I agree that the behavior is a
little surprising.

We could:
- have the value of -config replace ./swift.conf in the search path,
while keeping everything else unchanged
- not have ./swift.conf in the search path
- keep it as is in 0.95, but print out a message that both are being
used

> * If you invoke a Swift script from another directory, does the config file
> in your current directory or the script directory take effect?  It seems
> counterintuitive to me that you would have to cd into the script directory
> when you're already explicitly giving the path to the Swift script.

In 0.95 it is my understanding from reading the code that it would be
the current directory. I suspect this is a simple oversight. Both the
"new configs" are young code, and these things happen.

>From what Mike was saying, and unless folks disagree, I would think that
it should use swift.conf in the swift script directory.

> * Is there a way I can temporarily disable a config file on the search
> path, beyond moving the file or overriding all of the keys?  If so, is this
> method discoverable to users?

Not with any of the pre-trunk schemes. With explicit inheritance it
would be a matter of commenting out an included file (and maybe
additionally adding includes to any chained includes from the skipped
file). I would say this is pretty obvious and therefore trivially
discoverable.

It could be possible to have the config search path be defined by the
user, in a spirit similar to $PATH, but that might be pushing things too
far.

> * If ./swift.conf is meant to correspond to a particular Swift script,
> would it make more sense to have the config name match the script name?

That's a thought. Maybe. The consequences would be that you would need
to re-name configuration files if you use them with another script and
you would need to have multiple copies/links if you have multiple swift
scripts in the same directory.

On the topic of includes, what do you think about specifying the config
as a pragma directly in the swift script?

Mihael




From wilde at anl.gov  Mon Jul 14 20:04:13 2014
From: wilde at anl.gov (Michael Wilde)
Date: Mon, 14 Jul 2014 20:04:13 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405360828.24817.6.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>	
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>	
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>	
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>	
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>	
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
Message-ID: <53C47E0D.6080609@anl.gov>


On 7/14/14, 1:00 PM, Mihael Hategan wrote:
> The first question is whether ~/.swift/swift.conf and ./swift.conf
> should be explicitly in the search path, since SWIFT_SITE_CONF can do
> both:
>
> export SWIFT_SITE_CONF=~/.swift/swift.conf
> export SWIFT_SITE_CONF=./swift.conf
This variable was intended specifically for shared site definitions, to 
be provided by the group that administers Swift on the site (eg, like 
the RCC team), for use by all Swift users at the location.

If its value can only be a single filename, it can not serve this 
site-wide role in addition to the two alternative roles you show above:  
personal settings (~/.swift/swift.conf), and run-specific settings 
(./swift.conf). This variable can only serve one of these three roles.

But I don't think that this is the first question.  The first question 
is, should we have a search path for conf settings, and if so, what 
should it be.

You indicated however that ~/.swift/swift.conf remains in use for 
personal settings. That's good.

So the sole difference in property search path between 0.95 RC6 and 
trunk is that trunk no longer reads a property file by a fixed name from 
the current directory in which swift is run.

0.94 had:

swift installation
user
-config
command-line options

0.95 had:

swift installation
site-wide (new)
user
script (./) (new)
-config
command line

trunk has:

swift installation
site-wide
user
-config
command line

So the first point to discuss is in fact do we retain or remove the 0.95 
notion that properties that apply to a specific script can be placed 
alongside the script with no additional arguments on the swift command. 
Because that is, from considerable experience, the way most users use 
swift. Ive been creating a .cf file and specifying "-config cf" for many 
many years. Its just a question of whether virtually every swift 
invocation will continue to need to specify -config or not.

If the argument is that the user might mis-spell the name of the 
./swift.conf file, and perform a run with differet properties than were 
intended, I think thats a weak argument for removing this feature.  The 
user could just as easily mis-spell ~/.swift/swift.conf. An emacs user 
could mispell ~/.emacs or ~/.bashrc and the intended file would also be 
silently ignored. But there's a strong precedent for implicitly assumed 
config filenames in a variety of locations, and I think ./swift.conf 
calls into that same category.

One option is to add a property that specifies whether a default 
per-script config file is read.  This would be typically then specified 
by the user (or the tutorial setup) in ~/.swift/swift.conf. E.g.:
   perScriptConfig: ./swift.conf

If we need to keep specifying -config, I can live with that.  I'm more 
concerned that, in swift tutorials, simple things be simple. I like the 
simplicity of saying "swift part06.swift".

- Mike



From wilde at anl.gov  Mon Jul 14 21:09:41 2014
From: wilde at anl.gov (Michael Wilde)
Date: Mon, 14 Jul 2014 21:09:41 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405373372.26058.30.camel@echo>
References: <53C2AD4B.9030605@anl.gov>
	<1405274479.16177.13.camel@echo>	<53C30217.20501@anl.gov>
	<1405294638.17861.33.camel@echo>	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>	<1405302915.18573.37.camel@echo>
	<53C33980.1070608@anl.gov>	<1405304745.18573.63.camel@echo>
	<53C34009.9010302@anl.gov>	<1405306957.18573.85.camel@echo>
	<1405360828.24817.6.camel@echo>	<1405368398.25594.7.camel@echo>	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
	<1405373372.26058.30.camel@echo>
Message-ID: <53C48D65.1000008@anl.gov>

I sent an earlier reply with having seen all the latest posts to this 
thread.
This reply tries to catch up - sorry!

On 7/14/14, 4:29 PM, Mihael Hategan wrote:
> Thanks. Inline...
>
> On Mon, 2014-07-14 at 15:57 -0500, Tim Armstrong wrote:
>> I think the implicit versus explicit thing is a matter of taste; that said,
>> as a personal preference it seems more elegant and convenient to have a
>> powerful but explicit include mechanism so that there's an easy and obvious
>> way to include alternative files at any stage in the process.
>> A few observations/questions:
>> * As a user I'd find it really surprising that the implicit ./swift.conf
>> still is in effect when I provided an explicit command line config.  I'm
>> not aware of any other systems that behave like this.
> That seems to be what happens in 0.95. I agree that the behavior is a
> little surprising.
>
> We could:
> - have the value of -config replace ./swift.conf in the search path,
> while keeping everything else unchanged
I agree - that would be good, and better than the current 0.95 behavior.

> - not have ./swift.conf in the search path
> - keep it as is in 0.95, but print out a message that both are being
> used
>
>> * If you invoke a Swift script from another directory, does the config file
>> in your current directory or the script directory take effect?  It seems
>> counterintuitive to me that you would have to cd into the script directory
>> when you're already explicitly giving the path to the Swift script.
> In 0.95 it is my understanding from reading the code that it would be
> the current directory. I suspect this is a simple oversight. Both the
> "new configs" are young code, and these things happen.
>
>  From what Mike was saying, and unless folks disagree, I would think that
> it should use swift.conf in the swift script directory.
I think it would be better to expect swift.conf in the current directory 
of the swift command.
The script may be remote; the cwd is where the user will be "working" 
and creating run directories, and its where the properties for the 
execution of the requested script would more conveniently placed.
>
>> * Is there a way I can temporarily disable a config file on the search
>> path, beyond moving the file or overriding all of the keys?  If so, is this
>> method discoverable to users?
> Not with any of the pre-trunk schemes. With explicit inheritance it
> would be a matter of commenting out an included file (and maybe
> additionally adding includes to any chained includes from the skipped
> file). I would say this is pretty obvious and therefore trivially
> discoverable.
>
> It could be possible to have the config search path be defined by the
> user, in a spirit similar to $PATH, but that might be pushing things too
> far.
We could add properties that you could specify in ~/.swift or higher: 
e.g., noSwiftConfFile, noSwiftScriptFile, etc.
(Ive not given any thought to these actual names, though). Ive suggested 
another property-affecting-property in a prior response, and these would 
have a similar role. At the risk of making the config mechanism still 
more complex for the purpose of making it more tailorable to individual 
preference.
>> * If ./swift.conf is meant to correspond to a particular Swift script,
>> would it make more sense to have the config name match the script name?
> That's a thought. Maybe. The consequences would be that you would need
> to re-name configuration files if you use them with another script and
> you would need to have multiple copies/links if you have multiple swift
> scripts in the same directory.
I think not, again for the reason above. ./swift.conf is more specific 
to the task the user istrying to perform than to the specific script 
being used.
>
> On the topic of includes, what do you think about specifying the config
> as a pragma directly in the swift script?
I could see that being useful on occasion to bind specific behavior to a 
script, but its yet another set of specs and philosophies to wrestle with.

I'd vote for holding off on discussion of this pragma aspect until we 
get to a consensus on the current points in this thread.

so we have this implicit search path by default:

releaseConf=$SWIFT_HOME/etc/swift.conf
siteConf=$SWIFT_SITE_CONF
userConf=$HOME/.swift/swift.conf
runConf=$PWD/swift.conf

and these command line options:

-config
-specific-properties

To allow the default search path to be overridden on the command line, 
we can permit:

-config file1:file2:...:fileN  # use these config files to replace runConf
-reconfig file:file2:...:fileN # use these config files to replace all 
implicit config (including Swift's built-in defaults)

-noReleaseConf  -noSiteConf -noUserConf -noRunConf # use these flags to 
remove implicit config files from config path

If in addition you leave in the include facility, I think all options 
are possible, and there's a reasonable rationale and logic to the process.

It does make testing quite a challenge; documentation should not be too 
bad provided we dont uncover weird cases.

Usability should be good, if we mostly leave things alone and use either 
./swift.conf or -config oneFile

Mike

>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From hategan at mcs.anl.gov  Mon Jul 14 21:15:21 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 14 Jul 2014 19:15:21 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C47E0D.6080609@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<53C47E0D.6080609@anl.gov>
Message-ID: <1405390521.28246.41.camel@echo>

On Mon, 2014-07-14 at 20:04 -0500, Michael Wilde wrote:
> On 7/14/14, 1:00 PM, Mihael Hategan wrote:
> > The first question is whether ~/.swift/swift.conf and ./swift.conf
> > should be explicitly in the search path, since SWIFT_SITE_CONF can do
> > both:
> >
> > export SWIFT_SITE_CONF=~/.swift/swift.conf
> > export SWIFT_SITE_CONF=./swift.conf
> This variable was intended specifically for shared site definitions, to 
> be provided by the group that administers Swift on the site (eg, like 
> the RCC team), for use by all Swift users at the location.

Ok. I'm assuming the intent was for this to always be loaded by swift if
the environment variable is present, no matter what the user says in
other places. It could be disabled if really needed by unsetting the
env. var.

> 
> If its value can only be a single filename, it can not serve this 
> site-wide role in addition to the two alternative roles you show above:  
> personal settings (~/.swift/swift.conf), and run-specific settings 
> (./swift.conf). This variable can only serve one of these three roles.

Right. Although the user could include a site wide configuration, but
that would not closely match the intent of it (i.e. a site-imposed
setting). I think there is agreement here that this should stay, and
should be loaded after etc/swift.conf.

> 
> But I don't think that this is the first question.  The first question 
> is, should we have a search path for conf settings, and if so, what 
> should it be.

I think that there is agreement that at least swift/etc/swift.conf (with
sufficient information to allow simple local execution) and
argv["-config"] and then SWIFT_SITE_CONF are all needed, so I think that
answers the question of whether there should be a search path.

> 
> You indicated however that ~/.swift/swift.conf remains in use for 
> personal settings. That's good.

Sorry. Not currently. This was one of the things that was under
question. So should we still have it? Should we instead leave it as an
option to be included by the user from a more specific configuration
file?

This was in a sense intended as a site-wide configuration (for a given
user), so I felt, perhaps wrongly, that it did overlap with
SWIFT_SITE_CONF. 

[...]
> 
> So the first point to discuss is in fact do we retain or remove the 0.95 
> notion that properties that apply to a specific script can be placed 
> alongside the script with no additional arguments on the swift command. 
> Because that is, from considerable experience, the way most users use 
> swift. Ive been creating a .cf file and specifying "-config cf" for many 
> many years. Its just a question of whether virtually every swift 
> invocation will continue to need to specify -config or not.

3x the problem until recently: sites, tc, and config.

I personally didn't care. I wrote a simple run script, so I also didn't
have to type the swift script name every time. Though in both cases, I
just type once, and then use the command line history.

> 
> If the argument is that the user might mis-spell the name of the 
> ./swift.conf file, and perform a run with differet properties than were 
> intended, I think thats a weak argument for removing this feature.  The 
> user could just as easily mis-spell ~/.swift/swift.conf.

That's correct. Though it would be a less frequent problem, since it is
one file vs. many. Same for ./.bashrc, ./.emacs. I agree that ~/. files
are standard practice. Implicit files in the current directory, less so,
with, perhaps the exception of make.

However the argument was an example that was trying to illustrate that
silent automated choices are prone to subtle yet hard to troubleshoot
problems. The benefits may outweigh the risk. I don't really know. I'd
generally go with safety if it's only slightly less costly than the lack
of it. But it's personal preference.

>  An emacs user 
> could mispell ~/.emacs or ~/.bashrc and the intended file would also be 
> silently ignored. But there's a strong precedent for implicitly assumed 
> config filenames in a variety of locations, and I think ./swift.conf 
> calls into that same category.

> 
> One option is to add a property that specifies whether a default 
> per-script config file is read.  This would be typically then specified 
> by the user (or the tutorial setup) in ~/.swift/swift.conf. E.g.:
>    perScriptConfig: ./swift.conf

This is easy to do by >include ${env.PWD}/swift.conf< in
~/.swift/swift.conf, with no addition of a new property.

But see Tim's comments about including it from PWD vs. where the swift
script is.

> 
> If we need to keep specifying -config, I can live with that.  I'm more 
> concerned that, in swift tutorials, simple things be simple. I like the 
> simplicity of saying "swift part06.swift".

I think both etc/switf.conf and SWIFT_SITE_CONF are suitable for this.
This is not a statement against ./swift.conf, just an observation.

Mihael



From hategan at mcs.anl.gov  Mon Jul 14 21:56:17 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 14 Jul 2014 19:56:17 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C48D65.1000008@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
	<1405373372.26058.30.camel@echo> <53C48D65.1000008@anl.gov>
Message-ID: <1405392977.28246.62.camel@echo>

On Mon, 2014-07-14 at 21:09 -0500, Michael Wilde wrote:
[...]
> We could add properties that you could specify in ~/.swift or higher: 
> e.g., noSwiftConfFile, noSwiftScriptFile, etc.

I believe that there is a bit of technical difficulty there. A loaded
configuration is only fully defined when all the files have been loaded
(to allow all overrides to happen). These options would only be
available after all the files have been read, at which point the
decision of what was read had already been made.

It may be possible to define specific semantics for these options. For
example that they would apply as soon as the line is parsed, but,
again... technically difficult and the different classes of properties
raise some flags in my head.

[...]
> 
> so we have this implicit search path by default:
> 
> releaseConf=$SWIFT_HOME/etc/swift.conf
> siteConf=$SWIFT_SITE_CONF
> userConf=$HOME/.swift/swift.conf

While we're at it, but not worth a long discussion, should we have a
directory (.swift) for what is essentially a single file instead of
~/.swift.conf?

> runConf=$PWD/swift.conf
> 
> and these command line options:
> 
> -config
> -specific-properties

OK. Sounds reasonable. Combined with logging of the list of files used,
and the final property list with location for each one of them, this
should make troubleshooting easy.

I'd say -listconfig should only list the files used and state that a
full list of properties is available in the log file. This is to make
the output of -listconfig manageable and take care of the case where
users would not have otherwise checked the log.

> 
> To allow the default search path to be overridden on the command line, 
> we can permit:
> 
> -config file1:file2:...:fileN  # use these config files to replace runConf

This is equivalent to including file2...fileN from file1 . I'd vote
against having features that do what can be reasonably done by existing
features.

> -reconfig file:file2:...:fileN # use these config files to replace all 
> implicit config (including Swift's built-in defaults)

I think that this can also be achieved by a combination of -no*Conf and
-config + includes, so maybe we want to leave it at that? Makes both
testing and documentation simpler.

> 
> -noReleaseConf  -noSiteConf -noUserConf -noRunConf # use these flags to 
> remove implicit config files from config path

I think these address Tim's question about disabling configs quite well.

[...]

Mihael



From iraicu at cs.iit.edu  Tue Jul 15 09:19:04 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Tue, 15 Jul 2014 09:19:04 -0500
Subject: [Swift-devel] Call for Papers: IEEE Transactions on Cloud Computing
 - Special Issue on Scientific Cloud Computing (deadline Jul 31, 2014)
Message-ID: <53C53858.9000904@cs.iit.edu>

Dear colleagues,

This is just a friendly reminder about the upcoming deadline (July 31st, 
2014) for the special issue on Scientific Cloud Computing.

-------------------------------------------------------------------------------

Call for Papers

IEEE Transactions on Cloud Computing
Special Issue on Scientific Cloud Computing
http://datasys.cs.iit.edu/events/ScienceCloud2014-TCC/  

-------------------------------------------------------------------------------
IMPORTANT DATES

Paper Submissions Due: July 31, 2014
First Round Decision: September 30,2014
Major Revisions Due (if necessary): October 31, 2014
Final Decision: December 1, 2014
Journal Publication: TBD

-------------------------------------------------------------------------------
OVERVIEW

Computational and Data-Driven Sciences have become the third and fourth pillar
of scientific discovery in addition to experimental and theoretical sciences.
Scientific Computing has already begun to change how science is done, enabling
scientific breakthroughs through new kinds of experiments that would have been
impossible only a decade ago. It is the key to solving "grand challenges" in
many domains and providing breakthroughs in new knowledge, and it comes in many
shapes and forms: high-performance computing (HPC) which is heavily focused on
compute-intensive applications; high-throughput computing (HTC) which focuses
on using many computing resources over long periods of time to accomplish its
computational tasks; many-task computing (MTC) which aims to bridge the gap
between HPC and HTC by focusing on using many resources over short periods of
time; and data-intensive computing which is heavily focused on data
distribution, data-parallel execution, and harnessing data locality by
scheduling of computations close to the data. Today's "Big Data" trend is
generating datasets that are increasing exponentially in both complexity and
volume, making their analysis, archival, and sharing one of the grand
challenges of the 21st century. Not surprisingly, it becomes increasingly
difficult to design and operate large scale systems capable of addressing these
grand challenges.

This journal Special Issue on Scientific Cloud Computing in the IEEE
Transaction on Cloud Computing will provide the scientific community a
dedicated forum for discussing new research, development, and deployment
efforts in running these kinds of scientific computing workloads on Cloud
Computing infrastructures. This special issue will focus on the use of
cloud-based technologies to meet new compute-intensive and data-intensive
scientific challenges that are not well served by the current supercomputers,
grids and HPC clusters. The special issue will aim to address questions such
as: What architectural changes to the current cloud frameworks (hardware,
operating systems, networking and/or programming models) are needed to support
science? Dynamic information derived from remote instruments and coupled
simulation, and sensor ensembles that stream data for real-time analysis are
important emerging techniques in scientific and cyber-physical engineering
systems. How can cloud technologies enable and adapt to these new scientific
approaches dealing with dynamism? How are scientists using clouds? Are there
scientific HPC/HTC/MTC workloads that are suitable candidates to take advantage
of emerging cloud computing resources with high efficiency? Commercial public
clouds provide easy access to cloud infrastructure for scientists. What are the
gaps in commercial cloud offerings and how can they be adapted for running
existing and novel eScience applications? What benefits exist by adopting the
cloud model, over clusters, grids, or supercomputers? What factors are limiting
clouds use or would make them more usable/efficient?

-------------------------------------------------------------------------------
TOPICS

The topics of interest are, but not limited to, the application of Cloud in
scientific applications:

? Scientific application cases studies on Clouds
? Performance evaluation of Cloud technologies
? Fault tolerance and reliability in cloud systems
? Data-intensive workloads and tools on Clouds
? Programming models such as Map-Reduce
? Storage cloud architectures
? I/O and Data management in the Cloud
? Workflow and resource management in the Cloud
? NoSQL databases for scientific applications
? Data streaming and dynamic applications on Clouds
? Dynamic resource provisioning
? Many-Task Computing in the Cloud
? Application of cloud concepts in HPC environments
? Virtualized High performance parallel file systems
? Virtualized high performance I/O networks
? Virtualization and its Impact on Applications
? Distributed Operating Systems
? Many-core computing and accelerators in the Cloud
? Cloud security


-------------------------------------------------------------------------------
SUBMISSION INSTRUCTIONS
  
Authors are invited to submit papers with unpublished, original work to the
IEEE Transactions on Cloud Computing, Special Issue on Scientific Cloud
Computing. If the paper is extended from a workshop or conference paper, it
must contain at least 50% new material with "brand" new ideas and results. The
papers should not be longer than 14 double column pages in the IEEE TCC format.
Papers should be submitted directly to TCC at
https://mc.manuscriptcentral.com/tcc-cs, and "SI-ScienceCloud" should be
selected.

-------------------------------------------------------------------------------
ORGANIZERS

? Kate Keahey, University of Chicago & Argonne National Laboratory, USA
? Ioan Raicu, Illinois Institute of Technology & Argonne National Lab., USA
? Kyle Chard, University of Chicago & Argonne National Laboratory, USA
? Bogdan Nicolae, IBM Research, Ireland


-------------------------------------------------------------------------------
CONTACT

Email:sciencecloud2014-tcc-editors at datasys.cs.iit.edu  <mailto:sciencecloud2014-tcc-editors at datasys.cs.iit.edu>
Website:http://datasys.cs.iit.edu/events/ScienceCloud2014-TCC/

----------------------

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140715/dbe1e773/attachment.html>

From hategan at mcs.anl.gov  Wed Jul 16 01:04:12 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 15 Jul 2014 23:04:12 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405392977.28246.62.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
	<1405373372.26058.30.camel@echo> <53C48D65.1000008@anl.gov>
	<1405392977.28246.62.camel@echo>
Message-ID: <1405490652.15732.23.camel@echo>

Hi,

I'll be in New York for the remainder of the week and returning Monday
(21st). I might sporadically work on this, but in all likelihood commits
will come after that. However, I wanted so sync on the latest spec:

* releaseConf=$SWIFT_HOME/etc/swift.conf
siteConf=$SWIFT_SITE_CONF (if $SWIFT_SITE_CONF defined)
userConf=$HOME/.swift/swift.conf (if present)
runConf=$PWD/swift.conf (if present)
-config

* The configs should be chained automatically.

* Documentation updated, in particular copying and updating the 0.95
section on the search path and documenting swift-convert-config and how
to migrate from pre-0.95 files. The tool should also be updated to
accept and convert 0.95 configs.

* -noReleaseConf, -noSiteConf, -noUserConf, -noRunConf command-line
flags to be added. Please comment on the equivalent properties and the
difficulty in implementing them. If you feel it's important, it might be
worth conquering those difficulties.

* line numbers for properties in -listconfig and log.

Things that might still merit some discussion:

* -listconfig <files|full> (this is a proposal to add a parameter to
-listconfig that specifies whether only a list of files or a list of
files and all options should be printed).

* Also please comment on the -config file vs. -config file1:file2... and
the need for -reconfig. I can't quite tell whether these were meant to
be discussed or not, and if yes, where we are with that.

- another question is whether we want properties (in -listconfig and
log) to be printed as a flat set of dotted properties or grouped. In
other words:

site.one.workDirectory = ..
site.one.staging = ...

or 

site.one {
  workDirectory = ...
  staging = ...
}

Mihael





From yadunand at uchicago.edu  Wed Jul 16 01:17:39 2014
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Wed, 16 Jul 2014 01:17:39 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405490652.15732.23.camel@echo>
References: <53C2AD4B.9030605@anl.gov>
	<1405274479.16177.13.camel@echo>	<53C30217.20501@anl.gov>
	<1405294638.17861.33.camel@echo>	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>	<1405302915.18573.37.camel@echo>
	<53C33980.1070608@anl.gov>	<1405304745.18573.63.camel@echo>
	<53C34009.9010302@anl.gov>	<1405306957.18573.85.camel@echo>
	<1405360828.24817.6.camel@echo>	<1405368398.25594.7.camel@echo>	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>	<1405373372.26058.30.camel@echo>
	<53C48D65.1000008@anl.gov>	<1405392977.28246.62.camel@echo>
	<1405490652.15732.23.camel@echo>
Message-ID: <53C61903.3040903@uchicago.edu>

Hi Mihael,

Does the new config allow you to specify multiple sites in the 
swift.conf and select the site you want to run on from the swift 
commandline.
I searched the swift-trunk userguide, but did not see any reference to this.

The 0.95 tutorials, which I think are the cleanest our tutorials have 
been, let you select the site(s) you want to run on and the
swift.properties sets a default site(s) to use in case no commandline 
option is given for site

swift.properties:
# default site is local
site=local
site.beagle-remote {
}

site.local {

}

Thanks,
Yadu


On 07/16/2014 01:04 AM, Mihael Hategan wrote:
> Hi,
>
> I'll be in New York for the remainder of the week and returning Monday
> (21st). I might sporadically work on this, but in all likelihood commits
> will come after that. However, I wanted so sync on the latest spec:
>
> * releaseConf=$SWIFT_HOME/etc/swift.conf
> siteConf=$SWIFT_SITE_CONF (if $SWIFT_SITE_CONF defined)
> userConf=$HOME/.swift/swift.conf (if present)
> runConf=$PWD/swift.conf (if present)
> -config
>
> * The configs should be chained automatically.
>
> * Documentation updated, in particular copying and updating the 0.95
> section on the search path and documenting swift-convert-config and how
> to migrate from pre-0.95 files. The tool should also be updated to
> accept and convert 0.95 configs.
>
> * -noReleaseConf, -noSiteConf, -noUserConf, -noRunConf command-line
> flags to be added. Please comment on the equivalent properties and the
> difficulty in implementing them. If you feel it's important, it might be
> worth conquering those difficulties.
>
> * line numbers for properties in -listconfig and log.
>
> Things that might still merit some discussion:
>
> * -listconfig <files|full> (this is a proposal to add a parameter to
> -listconfig that specifies whether only a list of files or a list of
> files and all options should be printed).
>
> * Also please comment on the -config file vs. -config file1:file2... and
> the need for -reconfig. I can't quite tell whether these were meant to
> be discussed or not, and if yes, where we are with that.
>
> - another question is whether we want properties (in -listconfig and
> log) to be printed as a flat set of dotted properties or grouped. In
> other words:
>
> site.one.workDirectory = ..
> site.one.staging = ...
>
> or
>
> site.one {
>    workDirectory = ...
>    staging = ...
> }
>
> Mihael
>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140716/ac752633/attachment.html>

From wilde at anl.gov  Wed Jul 16 14:14:13 2014
From: wilde at anl.gov (Michael Wilde)
Date: Wed, 16 Jul 2014 14:14:13 -0500
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <1405490652.15732.23.camel@echo>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>	
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>	
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>	
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>	
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>	
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>	
	<1405368398.25594.7.camel@echo>	
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>	
	<1405373372.26058.30.camel@echo> <53C48D65.1000008@anl.gov>	
	<1405392977.28246.62.camel@echo> <1405490652.15732.23.camel@echo>
Message-ID: <53C6CF05.3020402@anl.gov>


On 7/16/14, 1:04 AM, Mihael Hategan wrote:
> Hi,
>
> I'll be in New York for the remainder of the week and returning Monday
> (21st). I might sporadically work on this, but in all likelihood commits
> will come after that. However, I wanted so sync on the latest spec:
>
> * releaseConf=$SWIFT_HOME/etc/swift.conf
> siteConf=$SWIFT_SITE_CONF (if $SWIFT_SITE_CONF defined)
> userConf=$HOME/.swift/swift.conf (if present)
> runConf=$PWD/swift.conf (if present)
> -config
...and then specific command line options.  Should all conf options be 
allowed, exactly as in the conf file? Or just all "scalar" options?
Should we move to a -D notation for specifying conf options on the 
command line, or leave as-is?
> * The configs should be chained automatically.
>
> * Documentation updated, in particular copying and updating the 0.95
> section on the search path and documenting swift-convert-config and how
> to migrate from pre-0.95 files. The tool should also be updated to
> accept and convert 0.95 configs.
>
> * -noReleaseConf, -noSiteConf, -noUserConf, -noRunConf command-line
> flags to be added. Please comment on the equivalent properties and the
> difficulty in implementing them. If you feel it's important, it might be
> worth conquering those difficulties.
Lets drop these. On re-consideration, they seem too complex.
>
> * line numbers for properties in -listconfig and log.
Good idea.
>
> Things that might still merit some discussion:
>
> * -listconfig <files|full> (this is a proposal to add a parameter to
> -listconfig that specifies whether only a list of files or a list of
> files and all options should be printed).
sounds ok
>
> * Also please comment on the -config file vs. -config file1:file2... and
> the need for -reconfig. I can't quite tell whether these were meant to
> be discussed or not, and if yes, where we are with that.
How about the following alternative to the above:

-config file replaces $PWD/swift.conf in the search path

-configpath file:file:file replaces the full search path with the 
specified path

SWIFT_CONF_PATH env var does the same thing as -configpath

This seems simpler.  Either accept the default search path, or specify 
your own.

There may be a few interactions between these that we would need to 
fully specify, but it seems like the right track.

For now, Im happy with just "-config file". The other two could be done 
at a later time.
>
> - another question is whether we want properties (in -listconfig and
> log) to be printed as a flat set of dotted properties or grouped. In
> other words:
>
> site.one.workDirectory = ..
> site.one.staging = ...
>
> or
>
> site.one {
>    workDirectory = ...
>    staging = ...
> }
The second is nicer; the first seems easier to code. Both are OK for now.

- Mike
>
> Mihael
>
>
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From yadunand at uchicago.edu  Wed Jul 16 17:36:40 2014
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Wed, 16 Jul 2014 17:36:40 -0500
Subject: [Swift-devel] Notes from trying new config
Message-ID: <53C6FE78.4000901@uchicago.edu>

Hi

I just tried to run some test scripts using trunk and these are my 
observations:

I copied the swift.conf file from the configuration section in the trunk 
userguide.

1. I got an error about lazy.errors=false. Found that lazy.errors has 
been replaces by lazyErrors. Minor doc mismatch here.
2. Then I got a swift.conf:2 missing mandatory property 
'site.local.workDirectory'
3. Then this -> swift.conf:2 missing mandatory property 
'site.local.filesystem.type'
     - This one is missing documentation as well.

I like that there are sensible error messages, but at the same time I 
feel that the config file has gotten a bit more verbose
that the 0.95 swift.properties. I really like the 0.95 feature where you 
could have several sites defined in your config file, and
then you choose what sites to run on from the commandline using 
-sites=<sites1, ...>. This makes the commandline very
simple and helps with tutorials and demos.

I could not get past this error once it looked like the config was fixed.

swift -config swift.conf sanity.swift
Swift trunk swift-r8021 cog-r4025 (cog modified locally)
RunID: run014
Could not start execution:
     Import of 'swift.k' failed, import @ sanity.kml, line: 5:
     Invalid node definition class: 
org.griphyn.vdl.karajan.lib.Mapping$Cons, def @ swift.k, line: 91:
     org.griphyn.vdl.karajan.lib.Mapping$Cons


-Yadu


From hategan at mcs.anl.gov  Wed Jul 16 22:23:00 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 16 Jul 2014 20:23:00 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C61903.3040903@uchicago.edu>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
	<1405373372.26058.30.camel@echo> <53C48D65.1000008@anl.gov>
	<1405392977.28246.62.camel@echo> <1405490652.15732.23.camel@echo>
	<53C61903.3040903@uchicago.edu>
Message-ID: <1405567380.18846.6.camel@echo>

On Wed, 2014-07-16 at 01:17 -0500, Yadu Nand Babuji wrote:
> Hi Mihael,
> 
> Does the new config allow you to specify multiple sites in the 
> swift.conf and select the site you want to run on from the swift 
> commandline.

Yes. Site declaration and site selection are separate, like in 0.95

> I searched the swift-trunk userguide, but did not see any reference to this.

Thanks for pointing that out. This is explained in Table 7 (General
Swift Options), but it clearly needs to be mentioned in the site
declaration section.

Mihael



From wilde at anl.gov  Fri Jul 18 13:57:47 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 18 Jul 2014 13:57:47 -0500
Subject: [Swift-devel] Error in 0.95 RC6 with wrapper.parameter.mode=files
Message-ID: <53C96E2B.50708@anl.gov>


In 0.95 RC6, I suspect we have a bug in handling 
wrapper.parameter.mode=files

WIth this set in swift.properties, a user Im working with is getting the 
error below.

Yadu, can you do a quick test to replicate this?

Thanks,

- Mike

---

java.lang.UnsupportedOperationException
     sys:file:write @ swift-int.k, line: 392
Caused by: java.lang.UnsupportedOperationException
     at 
org.globus.cog.karajan.analyzer.ChannelRef$Return.set(ChannelRef.java:88)
     at 
org.globus.cog.karajan.compiled.nodes.FileWrite.runBody(FileWrite.java:62)
     at 
org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:153)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at org.globus.cog.karajan.compiled.nodes.If.run(If.java:165)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.grid.AllocateHost.runBody(AllocateHost.java:78)
     at 
org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:153)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.user.Function.runBody(Function.java:99)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.runBody(InvocationWrapper.java:246)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.run(InvocationWrapper.java:69)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at org.globus.cog.karajan.compiled.nodes.Try.run(Try.java:52)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.RestartOnErrorNode.runBody(RestartOnErrorNode.java:62)
     at 
org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:153)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at org.griphyn.vdl.karajan.lib.Throttled.run(Throttled.java:62)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at org.globus.cog.karajan.compiled.nodes.Try.run(Try.java:52)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at org.globus.cog.karajan.compiled.nodes.If.run(If.java:165)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at org.globus.cog.karajan.compiled.nodes.If.run(If.java:165)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.user.Function.runBody(Function.java:99)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.runBody(InvocationWrapper.java:246)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.run(InvocationWrapper.java:69)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.user.Function.runBody(Function.java:99)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.runBody(InvocationWrapper.java:246)
     at 
org.globus.cog.karajan.compiled.nodes.user.InvocationWrapper.run(InvocationWrapper.java:69)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:87)
     at 
org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22)
     at 
org.griphyn.vdl.karajan.lib.ThrottledParallelFor$1.run(ThrottledParallelFor.java:196)
     at k.thr.LWThread.run(LWThread.java:228)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:744)


From wozniak at mcs.anl.gov  Fri Jul 18 16:19:04 2014
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Fri, 18 Jul 2014 16:19:04 -0500
Subject: [Swift-devel] Swift/T WWW to swift-lang.org
Message-ID: <53C98F48.2070805@mcs.anl.gov>

Hi
     I brought the Swift/T content into the CI WWW directories.  A 
preview is here:

http://web.ci.uchicago.edu/~wozniak/swift-www/main

     If there are no issues, I will post this to the live site.

     Justin

-- 
Justin M Wozniak




From wilde at anl.gov  Fri Jul 18 16:32:09 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 18 Jul 2014 16:32:09 -0500
Subject: [Swift-devel] Swift/T WWW to swift-lang.org
In-Reply-To: <53C98F48.2070805@mcs.anl.gov>
References: <53C98F48.2070805@mcs.anl.gov>
Message-ID: <53C99259.4030508@anl.gov>

This looks great, Justin.  I'd go ahead and post.

It might be handy to also post a link to the Leaf-writers guide from the 
main page (a 3rd blue button?) in addition to the link to that guide 
from the Leaf section of the User Guide.

- Mike

On 7/18/14, 4:19 PM, Justin M Wozniak wrote:
> Hi
>       I brought the Swift/T content into the CI WWW directories.  A
> preview is here:
>
> http://web.ci.uchicago.edu/~wozniak/swift-www/main
>
>       If there are no issues, I will post this to the live site.
>
>       Justin
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From wilde at anl.gov  Fri Jul 18 16:37:57 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 18 Jul 2014 16:37:57 -0500
Subject: [Swift-devel] Swift/T WWW to swift-lang.org
In-Reply-To: <53C99259.4030508@anl.gov>
References: <53C98F48.2070805@mcs.anl.gov> <53C99259.4030508@anl.gov>
Message-ID: <53C993B5.1090204@anl.gov>

Also, are we ready to post (at least prelim) main-wrapping info, 
possible as a section of the Leaf Writers Guide?

- Mike

On 7/18/14, 4:32 PM, Michael Wilde wrote:
> This looks great, Justin.  I'd go ahead and post.
>
> It might be handy to also post a link to the Leaf-writers guide from the
> main page (a 3rd blue button?) in addition to the link to that guide
> from the Leaf section of the User Guide.
>
> - Mike
>
> On 7/18/14, 4:19 PM, Justin M Wozniak wrote:
>> Hi
>>        I brought the Swift/T content into the CI WWW directories.  A
>> preview is here:
>>
>> http://web.ci.uchicago.edu/~wozniak/swift-www/main
>>
>>        If there are no issues, I will post this to the live site.
>>
>>        Justin
>>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From wilde at anl.gov  Fri Jul 18 16:41:49 2014
From: wilde at anl.gov (Michael Wilde)
Date: Fri, 18 Jul 2014 16:41:49 -0500
Subject: [Swift-devel] Swift/T WWW to swift-lang.org
In-Reply-To: <53C993B5.1090204@anl.gov>
References: <53C98F48.2070805@mcs.anl.gov> <53C99259.4030508@anl.gov>
	<53C993B5.1090204@anl.gov>
Message-ID: <53C9949D.2060507@anl.gov>

Also feel free to make the link to Swift/T in the main page more prominent.

Where it says:


      What's New


          NEXT GENERATION SWIFT

*Swift/T*runs millions of tasks per second on supercomputers.

Minimally, "Swift/T" above can be made into a nice blue link (to 
/Swift-T, same as "learn more" below that text)

- Mike




On 7/18/14, 4:37 PM, Michael Wilde wrote:
> Also, are we ready to post (at least prelim) main-wrapping info,
> possible as a section of the Leaf Writers Guide?
>
> - Mike
>
> On 7/18/14, 4:32 PM, Michael Wilde wrote:
>> This looks great, Justin.  I'd go ahead and post.
>>
>> It might be handy to also post a link to the Leaf-writers guide from the
>> main page (a 3rd blue button?) in addition to the link to that guide
>> from the Leaf section of the User Guide.
>>
>> - Mike
>>
>> On 7/18/14, 4:19 PM, Justin M Wozniak wrote:
>>> Hi
>>>         I brought the Swift/T content into the CI WWW directories.  A
>>> preview is here:
>>>
>>> http://web.ci.uchicago.edu/~wozniak/swift-www/main
>>>
>>>         If there are no issues, I will post this to the live site.
>>>
>>>         Justin
>>>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140718/29a5c1e0/attachment.html>

From wilde at anl.gov  Sat Jul 19 10:05:50 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sat, 19 Jul 2014 10:05:50 -0500
Subject: [Swift-devel] {env.HOME} not working in sites file in 0.95-RC6 ?
Message-ID: <53CA894E.7020308@anl.gov>

It seems like, in 0.95-RC6,  "{ }" substitution in sites.xml does not work.

A run with this fails:

<workdirectory>{env.HOME}/swiftwork</workdirectory>

while this works:

<workdirectory>/clhome/WILDE/swiftwork</workdirectory>

I'll need to isolate a simple test case, and search bugzilla, but was 
wondering if this is a known issue?

Is it tested in the test suite?

Thanks,

- Mike

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From davidkelly at uchicago.edu  Sat Jul 19 13:01:34 2014
From: davidkelly at uchicago.edu (David Kelly)
Date: Sat, 19 Jul 2014 13:01:34 -0500
Subject: [Swift-devel] {env.HOME} not working in sites file in 0.95-RC6 ?
In-Reply-To: <53CA894E.7020308@anl.gov>
References: <53CA894E.7020308@anl.gov>
Message-ID: <CA+_+Ey8_yDBkq_2j9XBVxNAsUPhoS775drc4xE_jKxnC1s+1TQ@mail.gmail.com>

Hi Mike,

Environment variables in sites.xml no longer works in 0.95. There was a
brief discussion about it in
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=423.

The test suite uses sed to replace {env.*} values, so I don't think it gets
tested directly.


On Sat, Jul 19, 2014 at 10:05 AM, Michael Wilde <wilde at anl.gov> wrote:

> It seems like, in 0.95-RC6,  "{ }" substitution in sites.xml does not work.
>
> A run with this fails:
>
> <workdirectory>{env.HOME}/swiftwork</workdirectory>
>
> while this works:
>
> <workdirectory>/clhome/WILDE/swiftwork</workdirectory>
>
> I'll need to isolate a simple test case, and search bugzilla, but was
> wondering if this is a known issue?
>
> Is it tested in the test suite?
>
> Thanks,
>
> - Mike
>
> --
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140719/c4c426e2/attachment.html>

From wilde at anl.gov  Sat Jul 19 13:16:32 2014
From: wilde at anl.gov (Michael Wilde)
Date: Sat, 19 Jul 2014 13:16:32 -0500
Subject: [Swift-devel] {env.HOME} not working in sites file in 0.95-RC6 ?
In-Reply-To: <CA+_+Ey8_yDBkq_2j9XBVxNAsUPhoS775drc4xE_jKxnC1s+1TQ@mail.gmail.com>
References: <53CA894E.7020308@anl.gov>
	<CA+_+Ey8_yDBkq_2j9XBVxNAsUPhoS775drc4xE_jKxnC1s+1TQ@mail.gmail.com>
Message-ID: <53CAB600.2070607@anl.gov>

Thanks, David. Indeed, that ticket clarifies the situation.

I see that 0.95 config allows eg "workdir=/scratch/midway/$USER/work" in 
the new-format config files, and that trunk/0.96 config allows ${env.PATH}

This will affect users of 0.95 that had { } expansion in their 0.94 
legacy xml-format sites files (which is the situation Im dealing with 
now).  But we can work around this with documentation and support.

- Mike

On 7/19/14, 1:01 PM, David Kelly wrote:
> Hi Mike,
>
> Environment variables in sites.xml no longer works in 0.95. There was 
> a brief discussion about it in 
> https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=423.
>
> The test suite uses sed to replace {env.*} values, so I don't think it 
> gets tested directly.
>
>
> On Sat, Jul 19, 2014 at 10:05 AM, Michael Wilde <wilde at anl.gov 
> <mailto:wilde at anl.gov>> wrote:
>
>     It seems like, in 0.95-RC6,  "{ }" substitution in sites.xml does
>     not work.
>
>     A run with this fails:
>
>     <workdirectory>{env.HOME}/swiftwork</workdirectory>
>
>     while this works:
>
>     <workdirectory>/clhome/WILDE/swiftwork</workdirectory>
>
>     I'll need to isolate a simple test case, and search bugzilla, but was
>     wondering if this is a known issue?
>
>     Is it tested in the test suite?
>
>     Thanks,
>
>     - Mike
>
>     --
>     Michael Wilde
>     Mathematics and Computer Science          Computation Institute
>     Argonne National Laboratory               The University of Chicago
>
>     _______________________________________________
>     Swift-devel mailing list
>     Swift-devel at ci.uchicago.edu <mailto:Swift-devel at ci.uchicago.edu>
>     https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140719/2b6a4b2e/attachment.html>

From wozniak at mcs.anl.gov  Mon Jul 21 10:27:17 2014
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Mon, 21 Jul 2014 10:27:17 -0500
Subject: [Swift-devel] Swift/T WWW to swift-lang.org
In-Reply-To: <53C993B5.1090204@anl.gov>
References: <53C98F48.2070805@mcs.anl.gov> <53C99259.4030508@anl.gov>
	<53C993B5.1090204@anl.gov>
Message-ID: <53CD3155.3020206@mcs.anl.gov>


Yes- we'll add it as another example.

On 7/18/2014 4:37 PM, Michael Wilde wrote:
> Also, are we ready to post (at least prelim) main-wrapping info,
> possible as a section of the Leaf Writers Guide?
>
> - Mike
>
> On 7/18/14, 4:32 PM, Michael Wilde wrote:
>> This looks great, Justin.  I'd go ahead and post.
>>
>> It might be handy to also post a link to the Leaf-writers guide from the
>> main page (a 3rd blue button?) in addition to the link to that guide
>> from the Leaf section of the User Guide.
>>
>> - Mike
>>
>> On 7/18/14, 4:19 PM, Justin M Wozniak wrote:
>>> Hi
>>>         I brought the Swift/T content into the CI WWW directories.  A
>>> preview is here:
>>>
>>> http://web.ci.uchicago.edu/~wozniak/swift-www/main
>>>
>>>         If there are no issues, I will post this to the live site.
>>>
>>>         Justin
>>>

-- 
Justin M Wozniak



From iraicu at cs.iit.edu  Wed Jul 23 16:23:56 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Wed, 23 Jul 2014 16:23:56 -0500
Subject: [Swift-devel] CFP: 1st International Workshop on Collaborative
 methodologies to Accelerate Scientific Knowledge discovery in big data
 (CASK) 2014 @ IEEE BigData 2014
Message-ID: <53D027EC.70108@cs.iit.edu>

Call for Papers

-------------------------------------------------------------------------------
1st International Workshop on Collaborative methodologies to Accelerate
Scientific Knowledge discovery in big data (CASK) 2014

Oct 27-30, 2014, Washington DC, US

In conjunction with:
2014 IEEE International Conference on Big Data (IEEE BigData 2014)

         http://bigscientificdata.org/cask14/
===============================================================================

Introduction
-------------------------------------------------------------------------------
Big Data has become an increasingly important part of life, and has become a
common buzzword used to describe many aspects of data intensive computing. One
of the unique aspects that we see to this new age of science is in terms of new
methods to collaborate and share ideas, data, and services. Service Oriented
Architectures have become commonplace in Enterprise computing, but its role in
scientific data has been often underplayed. Commonly scientists will write
software for themselves, and occasionally share their programs with their
colleagues. As computing moves to new levels of performance, by using
accelerators and many cores, one must rethink how scientific codes are produced
and move to new frameworks which promote collaboration.

In this workshop we want researchers to discuss techniques, infrastructure,
science drivers, and new ways to promote this new way of computing for
scientific applications. We will want to address fundamental issues in
workflows on and off large-scale high performance systems, clouds, IPads, and
mobile devices. Our overall view is that we can accelerate the scientific
knowledge discovery process by embracing new technologies where researchers can
share codes, workflows, data, and ultimately knowledge. Topics of interest in
this workshop will span from methods to share all aspects of code, data, and
workflows.  We will investigate the topic of how do you share Big Data, when
data gets to extreme sizes.  What new services need to be developed in order to
promote Big Data for science and engineering aspects? How can we get
researchers across the globe to develop shareable code and participate in the
greater science community? Just like math was a common language that
researchers could share and everyone understand, what are the new pieces of
software which must be developed in order to ensure collaboration across the
end to end lifecycle of scientific data?  The workshop will provide a venue to
show what has worked across different communities, and how to bring
collaboration to new scientific communities.  The workshop will bring together
DOE and NSF researchers along with researchers in the enterprise to present
papers, along with give invited talks. We will also conclude with a panel with
many leading experts in the field. We will also feature one panel which will
include experts in scientific and enterprise Big Data.


Topics
Topics of interest include, but are not limited to:
-------------------------------------------------------------------------------
* Data at Rest (Storage)
* Data in Motion (Data Streaming)
* Storage Systems (Database, file systems)
* Resource Management
* Query and Search
* Acquisition, Integrating, Cleaning and Best Practice
* Privacy
* Provenance
* Algorithms
* Analytics
* Visualization
* Near Real-time Decision Making
* Data Fusion
* Workflows
* Programming Models (e.g. MapReduce, MPI, etc.)


Important Dates
-------------------------------------------------------------------------------
* Papers Due: September 1st, 2014
* Notification of Acceptance: September 20th, 2014
* Camera Ready Papers Due: October 5th, 2014


Paper Submission
-------------------------------------------------------------------------------
Authors are invited to submit papers electronically. Submitted manuscripts
should be structured as technical papers and may not exceed 6 letter size
(8.5 x 11) pages including figures, tables and references using the IEEE format
for conference proceedings (print area of 6-1/2 inches (16.51 cm) wide by 8-7/8
inches (22.51 cm) high, two-column format with columns 3-1/16 inches (7.85 cm)
wide with a 3/8 inch (0.81 cm) space between them, single-spaced 10-point Times
fully justified text). Submissions not conforming to these guidelines may be
returned without review. Authors should submit the manuscript in PDF format and
make sure that the file will print on a printer that uses letter size (8.5 x 11)
paper. The official language of the meeting is English. All manuscripts will be
reviewed and will be judged on correctness, originality, technical strength,
significance, quality of presentation, and interest and relevance to the
conference attendees. Papers conforming to the above guidelines can be
submitted through the CASK 2014 paper submission system
(https://cmt.research.microsoft.com/CASK2014).

Submitted papers must represent original unpublished research that is not
currently under review for any other conference or journal. Papers not
following these guidelines will be rejected without review and further action
may be taken, including (but not limited to) notifications sent to the heads of
the institutions of the authors and sponsors of the conference. Submissions
received after the due date, exceeding length limit, or not appropriately
structured may also not be considered. Authors may contact the conference PC
Chair for more information. The proceedings will be published through the IEEE
Computer Society Press, USA and will be made online through the IEEE Digital
Library. Selected papers from CASK 2014 will be invited to extend and submit to
the Special Issue on Many-Task Computing in the Cloud in the IEEE Transaction on
Cloud Computing (http://datasys.cs.iit.edu/events/TCC-MTC15/CFP_TCC-MTC15.pdf).


Chairs and Committees
-------------------------------------------------------------------------------
Workshop Co-Chairs:

* Chen Jin (Palantir Technology)
* Ioan Raicu (Illinois Institute of Technology)
* Scott Klasky (Oak Ridge National Laboratory)

Program Co-Chairs:

* Raju Vatsavai (North Carolina State Univ. & Oak Ridge National Lab.)
* Judy Qiu (Indiana University)
* George Ostrouchov (Oak Ridge National Laboratory)
* Tahsin Kurc (Stony Brook University)
* Daniel S. Katz (University of Chicago)
* Bogdan Nicolae (IBM Research)
* Doug Thain (University of Notre Dame)
* Josh Wills (Cloudera)
* Zhengzhang Chen (NEC Labs)
* Kunpeng Zhang (University of Illinois at Chicago)

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140723/dee0a91e/attachment.html>

From hategan at mcs.anl.gov  Thu Jul 24 18:36:33 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 24 Jul 2014 16:36:33 -0700
Subject: [Swift-devel] Notes from trying new config
In-Reply-To: <53C6FE78.4000901@uchicago.edu>
References: <53C6FE78.4000901@uchicago.edu>
Message-ID: <1406244993.3249.0.camel@echo>

Was there ever a reply from me on this? I have one in my "Sent" folder,
but I don't see it on the mailing list.

Mihael

On Wed, 2014-07-16 at 17:36 -0500, Yadu Nand Babuji wrote:
> Hi
> 
> I just tried to run some test scripts using trunk and these are my 
> observations:
> 
> I copied the swift.conf file from the configuration section in the trunk 
> userguide.
> 
> 1. I got an error about lazy.errors=false. Found that lazy.errors has 
> been replaces by lazyErrors. Minor doc mismatch here.
> 2. Then I got a swift.conf:2 missing mandatory property 
> 'site.local.workDirectory'
> 3. Then this -> swift.conf:2 missing mandatory property 
> 'site.local.filesystem.type'
>      - This one is missing documentation as well.
> 
> I like that there are sensible error messages, but at the same time I 
> feel that the config file has gotten a bit more verbose
> that the 0.95 swift.properties. I really like the 0.95 feature where you 
> could have several sites defined in your config file, and
> then you choose what sites to run on from the commandline using 
> -sites=<sites1, ...>. This makes the commandline very
> simple and helps with tutorials and demos.
> 
> I could not get past this error once it looked like the config was fixed.
> 
> swift -config swift.conf sanity.swift
> Swift trunk swift-r8021 cog-r4025 (cog modified locally)
> RunID: run014
> Could not start execution:
>      Import of 'swift.k' failed, import @ sanity.kml, line: 5:
>      Invalid node definition class: 
> org.griphyn.vdl.karajan.lib.Mapping$Cons, def @ swift.k, line: 91:
>      org.griphyn.vdl.karajan.lib.Mapping$Cons
> 
> 
> -Yadu




From hategan at mcs.anl.gov  Thu Jul 24 18:38:15 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 24 Jul 2014 16:38:15 -0700
Subject: [Swift-devel] Issues to resolve for the 0.96 config mechanism
In-Reply-To: <53C6CF05.3020402@anl.gov>
References: <53C2AD4B.9030605@anl.gov> <1405274479.16177.13.camel@echo>
	<53C30217.20501@anl.gov> <1405294638.17861.33.camel@echo>
	<CAC0jiV7-+a9b5GHroD4G+yZJLmE0-zd4Dez9m_T9=TH88_81JA@mail.gmail.com>
	<1405302915.18573.37.camel@echo> <53C33980.1070608@anl.gov>
	<1405304745.18573.63.camel@echo> <53C34009.9010302@anl.gov>
	<1405306957.18573.85.camel@echo> <1405360828.24817.6.camel@echo>
	<1405368398.25594.7.camel@echo>
	<CAC0jiV5XNkFDVr_PbmvNfw5jfSMc-BmamVkhSjHiAWhe-vrk2A@mail.gmail.com>
	<1405373372.26058.30.camel@echo> <53C48D65.1000008@anl.gov>
	<1405392977.28246.62.camel@echo> <1405490652.15732.23.camel@echo>
	<53C6CF05.3020402@anl.gov>
Message-ID: <1406245095.3249.2.camel@echo>

This should now be all committed to trunk. Except for SWIFT_CONF_PATH. I
noticed I forgot about it while replying to this email.

Mihael

On Wed, 2014-07-16 at 14:14 -0500, Michael Wilde wrote:
> On 7/16/14, 1:04 AM, Mihael Hategan wrote:
> > Hi,
> >
> > I'll be in New York for the remainder of the week and returning Monday
> > (21st). I might sporadically work on this, but in all likelihood commits
> > will come after that. However, I wanted so sync on the latest spec:
> >
> > * releaseConf=$SWIFT_HOME/etc/swift.conf
> > siteConf=$SWIFT_SITE_CONF (if $SWIFT_SITE_CONF defined)
> > userConf=$HOME/.swift/swift.conf (if present)
> > runConf=$PWD/swift.conf (if present)
> > -config
> ...and then specific command line options.  Should all conf options be 
> allowed, exactly as in the conf file? Or just all "scalar" options?
> Should we move to a -D notation for specifying conf options on the 
> command line, or leave as-is?
> > * The configs should be chained automatically.
> >
> > * Documentation updated, in particular copying and updating the 0.95
> > section on the search path and documenting swift-convert-config and how
> > to migrate from pre-0.95 files. The tool should also be updated to
> > accept and convert 0.95 configs.
> >
> > * -noReleaseConf, -noSiteConf, -noUserConf, -noRunConf command-line
> > flags to be added. Please comment on the equivalent properties and the
> > difficulty in implementing them. If you feel it's important, it might be
> > worth conquering those difficulties.
> Lets drop these. On re-consideration, they seem too complex.
> >
> > * line numbers for properties in -listconfig and log.
> Good idea.
> >
> > Things that might still merit some discussion:
> >
> > * -listconfig <files|full> (this is a proposal to add a parameter to
> > -listconfig that specifies whether only a list of files or a list of
> > files and all options should be printed).
> sounds ok
> >
> > * Also please comment on the -config file vs. -config file1:file2... and
> > the need for -reconfig. I can't quite tell whether these were meant to
> > be discussed or not, and if yes, where we are with that.
> How about the following alternative to the above:
> 
> -config file replaces $PWD/swift.conf in the search path
> 
> -configpath file:file:file replaces the full search path with the 
> specified path
> 
> SWIFT_CONF_PATH env var does the same thing as -configpath
> 
> This seems simpler.  Either accept the default search path, or specify 
> your own.
> 
> There may be a few interactions between these that we would need to 
> fully specify, but it seems like the right track.
> 
> For now, Im happy with just "-config file". The other two could be done 
> at a later time.
> >
> > - another question is whether we want properties (in -listconfig and
> > log) to be printed as a flat set of dotted properties or grouped. In
> > other words:
> >
> > site.one.workDirectory = ..
> > site.one.staging = ...
> >
> > or
> >
> > site.one {
> >    workDirectory = ...
> >    staging = ...
> > }
> The second is nicer; the first seems easier to code. Both are OK for now.
> 
> - Mike
> >
> > Mihael
> >
> >
> >
> 




From tim.g.armstrong at gmail.com  Thu Jul 24 22:13:30 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Thu, 24 Jul 2014 22:13:30 -0500
Subject: [Swift-devel] Notes from trying new config
In-Reply-To: <1406244993.3249.0.camel@echo>
References: <53C6FE78.4000901@uchicago.edu>
	<1406244993.3249.0.camel@echo>
Message-ID: <CAC0jiV4Gw9BmKzWiefQaR-eS4bPFftQkxK9BtksHRHKWYrHCkw@mail.gmail.com>

I don't think I saw one.

- Tim


On Thu, Jul 24, 2014 at 6:36 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Was there ever a reply from me on this? I have one in my "Sent" folder,
> but I don't see it on the mailing list.
>
> Mihael
>
> On Wed, 2014-07-16 at 17:36 -0500, Yadu Nand Babuji wrote:
> > Hi
> >
> > I just tried to run some test scripts using trunk and these are my
> > observations:
> >
> > I copied the swift.conf file from the configuration section in the trunk
> > userguide.
> >
> > 1. I got an error about lazy.errors=false. Found that lazy.errors has
> > been replaces by lazyErrors. Minor doc mismatch here.
> > 2. Then I got a swift.conf:2 missing mandatory property
> > 'site.local.workDirectory'
> > 3. Then this -> swift.conf:2 missing mandatory property
> > 'site.local.filesystem.type'
> >      - This one is missing documentation as well.
> >
> > I like that there are sensible error messages, but at the same time I
> > feel that the config file has gotten a bit more verbose
> > that the 0.95 swift.properties. I really like the 0.95 feature where you
> > could have several sites defined in your config file, and
> > then you choose what sites to run on from the commandline using
> > -sites=<sites1, ...>. This makes the commandline very
> > simple and helps with tutorials and demos.
> >
> > I could not get past this error once it looked like the config was fixed.
> >
> > swift -config swift.conf sanity.swift
> > Swift trunk swift-r8021 cog-r4025 (cog modified locally)
> > RunID: run014
> > Could not start execution:
> >      Import of 'swift.k' failed, import @ sanity.kml, line: 5:
> >      Invalid node definition class:
> > org.griphyn.vdl.karajan.lib.Mapping$Cons, def @ swift.k, line: 91:
> >      org.griphyn.vdl.karajan.lib.Mapping$Cons
> >
> >
> > -Yadu
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140724/a3ddeee1/attachment.html>

From yadudoc1729 at gmail.com  Thu Jul 24 23:12:28 2014
From: yadudoc1729 at gmail.com (Yadu Nand)
Date: Thu, 24 Jul 2014 23:12:28 -0500
Subject: [Swift-devel] Notes from trying new config
In-Reply-To: <CAC0jiV4Gw9BmKzWiefQaR-eS4bPFftQkxK9BtksHRHKWYrHCkw@mail.gmail.com>
References: <53C6FE78.4000901@uchicago.edu> <1406244993.3249.0.camel@echo>
	<CAC0jiV4Gw9BmKzWiefQaR-eS4bPFftQkxK9BtksHRHKWYrHCkw@mail.gmail.com>
Message-ID: <CANa904=rEGpTmxdfprz+08ANWvJ+PGVYW-ewkYn1KzDd9MDWpA@mail.gmail.com>

The issues I raised are mostly resolved:

1. I don't see warnings about lazy.errors / lazyErrors,
2. I think workDirectory errors were alright, and is better off as a
required option.
3. I got the filesystem.type options for docs.
The docs also now explain how to use -sites to pick the site from the
config.

Now that Mihael is on the thread, I'd like to point out that the nesting
and braces are excessive.
In case of execution options, I don't see why there has to be a separate
group under execution. Same is the case with
jobOptions nested under options. Here's the config I'm using as an example :

site.cloud {
    execution {
        type:"coaster"
        URL: "127.0.0.1"
        jobManager: "local:ec2-cloud"
        options {
            maxJobs: 10
            tasksPerNode: 4
            workerLoggingLevel: TRACE
            workerLoggingDirectory: /tmp
            jobOptions {
                ec2CredentialsFile:
${env.HOME}/.ssh/boto-test-credentials.csv
                ec2SecurityGroup: swift_security_group1
                ec2KeypairName: swift-test-pair
                ec2KeypairFile: ${env.HOME}/swift-test-pair.pem
                ec2WorkerImage: ami-23700813
                ec2WorkerType: t1.micro
            }
        }
    }

    # Latest trunk won't run without the next two:
    execution.retries: 2
    wrapperlog.always.transfer: False

    initialParallelTasks: 10
    maxParallelTasks: 20
    filesystem.type: swift
    workDirectory: /tmp/swift-trunk-test
    staging: "local"
    app.ALL {executable: "*"}
}

sites: cloud

Thanks,
Yadu



On Thu, Jul 24, 2014 at 10:13 PM, Tim Armstrong <tim.g.armstrong at gmail.com>
wrote:

> I don't think I saw one.
>
> - Tim
>
>
> On Thu, Jul 24, 2014 at 6:36 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>
>> Was there ever a reply from me on this? I have one in my "Sent" folder,
>> but I don't see it on the mailing list.
>>
>> Mihael
>>
>> On Wed, 2014-07-16 at 17:36 -0500, Yadu Nand Babuji wrote:
>> > Hi
>> >
>> > I just tried to run some test scripts using trunk and these are my
>> > observations:
>> >
>> > I copied the swift.conf file from the configuration section in the trunk
>> > userguide.
>> >
>> > 1. I got an error about lazy.errors=false. Found that lazy.errors has
>> > been replaces by lazyErrors. Minor doc mismatch here.
>> > 2. Then I got a swift.conf:2 missing mandatory property
>> > 'site.local.workDirectory'
>> > 3. Then this -> swift.conf:2 missing mandatory property
>> > 'site.local.filesystem.type'
>> >      - This one is missing documentation as well.
>> >
>> > I like that there are sensible error messages, but at the same time I
>> > feel that the config file has gotten a bit more verbose
>> > that the 0.95 swift.properties. I really like the 0.95 feature where you
>> > could have several sites defined in your config file, and
>> > then you choose what sites to run on from the commandline using
>> > -sites=<sites1, ...>. This makes the commandline very
>> > simple and helps with tutorials and demos.
>> >
>> > I could not get past this error once it looked like the config was
>> fixed.
>> >
>> > swift -config swift.conf sanity.swift
>> > Swift trunk swift-r8021 cog-r4025 (cog modified locally)
>> > RunID: run014
>> > Could not start execution:
>> >      Import of 'swift.k' failed, import @ sanity.kml, line: 5:
>> >      Invalid node definition class:
>> > org.griphyn.vdl.karajan.lib.Mapping$Cons, def @ swift.k, line: 91:
>> >      org.griphyn.vdl.karajan.lib.Mapping$Cons
>> >
>> >
>> > -Yadu
>>
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


-- 
Yadu Nand B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140724/b57fa3de/attachment.html>

From hategan at mcs.anl.gov  Thu Jul 24 23:50:26 2014
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 24 Jul 2014 21:50:26 -0700
Subject: [Swift-devel] Notes from trying new config
In-Reply-To: <CANa904=rEGpTmxdfprz+08ANWvJ+PGVYW-ewkYn1KzDd9MDWpA@mail.gmail.com>
References: <53C6FE78.4000901@uchicago.edu> <1406244993.3249.0.camel@echo>
	<CAC0jiV4Gw9BmKzWiefQaR-eS4bPFftQkxK9BtksHRHKWYrHCkw@mail.gmail.com>
	<CANa904=rEGpTmxdfprz+08ANWvJ+PGVYW-ewkYn1KzDd9MDWpA@mail.gmail.com>
Message-ID: <1406263826.6465.22.camel@echo>

On Thu, 2014-07-24 at 23:12 -0500, Yadu Nand wrote:
> The issues I raised are mostly resolved:
> 
> 1. I don't see warnings about lazy.errors / lazyErrors,
> 2. I think workDirectory errors were alright, and is better off as a
> required option.

I don't think that it being required was ever under discussion. It was
required, but you would get weird errors at run-time if it wasn't
specified, instead of when the sites file was loaded.

> 3. I got the filesystem.type options for docs.
> The docs also now explain how to use -sites to pick the site from the
> config.
> 
> Now that Mihael is on the thread, I'd like to point out that the nesting
> and braces are excessive.
> In case of execution options, I don't see why there has to be a separate
> group under execution. Same is the case with
> jobOptions nested under options.

you can always say "options.maxJobs: 10" if the braces are a problem.

The reason for "options" is that they create a logical grouping for
optional tunable things for a provider or for a job, but the important
thing about them is that they are provider-specific. I wanted to
separate them from things that apply in general. For example, from
quickly looking at the list of options in 0.95, could you tell whether
the "pe" option applies to a specific provider or all swift jobs? What
about "maxSubmitRate"?

We can discuss this I guess. I thought that the main reason these were
not separated in 0.95 was because they were following the flat structure
of the pre-0.95 sites file.

As for jobOptions, it was previously providerAttributes, which accepted
a string of comma-separated key-value pairs. jobOptions is simply
re-using the existing configuration syntax instead of having its own
syntax. Logically, like jobOptions, providerAttributes was a container
for various settings. 0.95 does not address this issue. Again, I'm happy
to discuss alternatives. Though, honestly, I don't see much of a
difference in typing effort between

providerAttributes="ec2WorkerImage=ami-23700813, ec2WorkerType=t1.micro"

and

jobOptions {ec2WorkerImage: ami-23700813, ec2WorkerType: t1.micro}

Mihael

>  Here's the config I'm using as an example :
> 
> site.cloud {
>     execution {
>         type:"coaster"
>         URL: "127.0.0.1"
>         jobManager: "local:ec2-cloud"
>         options {
>             maxJobs: 10
>             tasksPerNode: 4
>             workerLoggingLevel: TRACE
>             workerLoggingDirectory: /tmp
>             jobOptions {
>                 ec2CredentialsFile:
> ${env.HOME}/.ssh/boto-test-credentials.csv
>                 ec2SecurityGroup: swift_security_group1
>                 ec2KeypairName: swift-test-pair
>                 ec2KeypairFile: ${env.HOME}/swift-test-pair.pem
>                 ec2WorkerImage: ami-23700813
>                 ec2WorkerType: t1.micro
>             }
>         }
>     }
> 
>     # Latest trunk won't run without the next two:
>     execution.retries: 2
>     wrapperlog.always.transfer: False
> 
>     initialParallelTasks: 10
>     maxParallelTasks: 20
>     filesystem.type: swift
>     workDirectory: /tmp/swift-trunk-test
>     staging: "local"
>     app.ALL {executable: "*"}
> }
> 
> sites: cloud




From iraicu at cs.iit.edu  Fri Jul 25 11:38:58 2014
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Fri, 25 Jul 2014 11:38:58 -0500
Subject: [Swift-devel] CFP: International Symposium on Big Data Computing
	(BDC) 2014
Message-ID: <53D28822.1020605@cs.iit.edu>

Call for Papers

IEEE/ACM International Symposium on Big Data Computing (BDC) 2014
December 8-11, 2014, London, UK
http://www.cloudbus.org/bdc2014

In conjunction with:
7th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2014)
Sponsored by: IEEE Computer Society and ACM (Association for Computing Machinery)

Introduction
===============================================================================
Rapid advances in digital sensors, networks, storage, and computation along
with their availability at low cost is leading to the creation of huge
collections of data -- dubbed as Big Data. This data has the potential for
enabling new insights that can change the way business, science, and
governments deliver services to their consumers and can impact society as a
whole. This has led to the emergence of the Big Data Computing paradigm
focusing on sensing, collection, storage, management and analysis of data from
variety of sources to enable new value and insights.

To realize the full potential of Big Data Computing, we need to address
several challenges and develop suitable conceptual and technological solutions
for dealing them. These include life-cycle management of data, large-scale
storage, flexible processing infrastructure, data modelling, scalable machine
learning and data analysis algorithms, techniques for sampling and making
trade-off between data processing time and accuracy, and dealing with privacy
and ethical issues involved in data sensing, storage, processing, and actions.

The IEEE/ACM International Symposium on Big Data Computing (BDC) 2014 -- held
in conjunction with 7th IEEE/ACM International Conference on Utility and Cloud
Computing (UCC 2014), December 8-11, 2014, London, UK, aims at bringing
together international researchers, developers, policy makers, and users and
to provide an international forum to present leading research activities,
technical solutions, and results on a broad range of topics related to Big
Data Computing paradigms, platforms and their applications. The conference
features keynotes, technical presentations, posters, workshops, tutorials, as
well as competitions featuring live demonstrations.

Topics
===============================================================================
Topics of interest include, but are not limited to:
I. Big Data Science
* Analytics
* Algorithms for Big Data
* Energy-efficient Algorithms
* Big Data Search
* Big Data Acquisition, Integration, Cleaning, and Best Practices
* Visualization of Big Data

II. Big Data Infrastructures and Platforms
* Programming Systems
* Cyber-Infrastructure
* Performance evaluation
* Fault tolerance and reliability
* I/O and Data management
* Storage Systems (including file systems, NoSQL, and RDBMS)
* Resource management
* Many-Task Computing
* Many-core computing and accelerators

III. Big Data Security and Policy
* Management Policies
* Data Privacy
* Data Security
* Big Data Archival and Preservation
* Big Data Provenance

IV. Big Data Applications
* Scientific application cases studies on Cloud infrastructure
* Big Data Applications at Scale
* Experience Papers with Big Data Application Deployments
* Data streaming applications
* Big Data in Social Networks
* Healthcare Applications
* Enterprise Applications

IMPORTANT DATES
===============================================================================
* Papers Due:                   September 15th, 2014
* Notification of Acceptance:   October 15th, 2014
* Camera Ready Papers Due:      October 31st, 2014

PAPER SUBMISSION
===============================================================================
Authors are invited to submit papers electronically. Submitted manuscripts
should be structured as technical papers and may not exceed 10 letter size
(8.5 x 11) pages including figures, tables and references using the IEEE format
for conference proceedings (print area of 6-1/2 inches (16.51 cm) wide by 8-7/8
inches (22.51 cm) high, two-column format with columns 3-1/16 inches (7.85 cm)
wide with a 3/8 inch (0.81 cm) space between them, single-spaced 10-point Times
fully justified text). Submissions not conforming to these guidelines may be
returned without review. Authors should submit the manuscript in PDF format and
make sure that the file will print on a printer that uses letter size (8.5 x 11)
paper. The official language of the meeting is English. All manuscripts will be
reviewed and will be judged on correctness, originality, technical strength,
significance, quality of presentation, and interest and relevance to the
conference attendees. Papers conforming to the above guidelines can be
submitted through the BDC 2014 paper submission system
(https://www.easychair.org/conferences/?conf=bdc2014).

Submitted papers must represent original unpublished research that is not
currently under review for any other conference or journal. Papers not
following these guidelines will be rejected without review and further action
may be taken, including (but not limited to) notifications sent to the heads
of the institutions of the authors and sponsors of the conference. Submissions
received after the due date, exceeding length limit, or not appropriately
structured may also not be considered. Authors may contact the conference PC
Chair for more information. The proceedings will be published through the IEEE
Computer Society Press, USA and will be made online through the IEEE Digital
Library. Selected papers from BDC 2014 will be invited to extend and submit to
the Special Issue on Many-Task Computing in the Cloud in the IEEE Transaction
on Cloud Computing (http://datasys.cs.iit.edu/events/TCC-MTC15/CFP_TCC-MTC15.pdf)

CHAIRS & COMMITTEES
===============================================================================
General Co-Chairs:
* Rajkumar Buyya, University of Melbourne, Australia
* Divyakant Agrawal, University of California at Santa Barbara, USA

Program Co-Chairs:
* Ioan Raicu, Illinois Institute of Technology and Argonne National Lab., USA
* Manish Parashar, Rutgers, The State University of New Jersey, USA

Area Track Co-Chairs:
* Big Data Science
   o TBA
* Data Infrastructures and Platforms
   o Amy Apon, Clemson University, USA
   o Jiannong Cao, Honk Kong Polytechnic University
* Big Data Security and Policy
   o Bogdan Carbunar, Florida International University
* Big Data Applications
   o Dennis Gannon, Microsoft Research, USA

Cyber Chair
* Amir Vahid, University of Melbourne, Australia

Publicity Chairs
* Carlos Westphall, Federal University of Santa Catarina, Brazil
* Ching-Hsien Hsu, Chung Hua Univ., Taiwan & Tianjin Univ. of Technology, China
* Rong Ge, Marquette University, USA

Organizing Chair:
* Ashiq Anjum, University of Derby, UK


-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140725/4eb9a2ae/attachment.html>

From tim.g.armstrong at gmail.com  Thu Jul 31 19:08:25 2014
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Thu, 31 Jul 2014 19:08:25 -0500
Subject: [Swift-devel] Swift/K is now mirrored on Github
Message-ID: <CAC0jiV4wpRTfbjTswAzDJUu-3JX6WUF18_V59WvnEEk6H-TprA@mail.gmail.com>

I've just flipped the switch to publish a mirror of the Swift/K source on
github.

   https://github.com/swift-lang/swift-k

This repository merges the contents of both the Swift and CoG repositories
into a single tree.  You can now check out and build Swift/K more simply:

  https://github.com/swift-lang/swift-k.git
  cd swift-k
  ant redist

Please test it out, play around, and report any issues to me.

For now, the github repo is a read-only mirror of the Swift and CoG
subversion repositories.  Commits to these repositories will be
automatically committed to github within 15-20 minutes of the subversion
commit.

Commits to the old repositories maps to directories of the github repo as
follows:

  https://svn.code.sf.net/p/cogkit/svn/trunk/src/cog -> ./cogkit/
  https://svn.ci.uchicago.edu/svn/vdl2/trunk -> ./

We can automatically publish branches and tags, but I'll need to add them
to a list.  Let me know if there are any missing branches or tags that you
think should appear on github.

- Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20140731/e8fd1ddd/attachment.html>