From ketan at mcs.anl.gov  Fri May  1 10:09:01 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 1 May 2015 10:09:01 -0500
Subject: [Swift-devel] source file extension for t and k
Message-ID: <CAMUuvio1QU=QNWN+6hnyhtaK+rTeHiFm2yGYTNUX44MC+f=Jeg@mail.gmail.com>

Should the T and K Swift source files have different extension for easy
recognition?

--Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150501/542df1e6/attachment.html>

From iraicu at cs.iit.edu  Wed May  6 12:22:45 2015
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Wed, 06 May 2015 12:22:45 -0500
Subject: [Swift-devel] Call for Participation and Posters: ACM HPDC'15 in
 Portland, Oregon, June 15-19, 2015
Message-ID: <554A4DE5.8010203@cs.iit.edu>

*HPDC?15 - Call for Participation and Posters**
**http://www.hpdc.org/2015/**
*
/*NEWS:*//*
*//*- Poster/Abstract submission deadline: May 8th, 2015*//*
*//*- Early Registration deadline: May 18th, 2015*//*
*//*- Conference dates: June 15th-19th, 2015*//*
*//*    - Keynote: Allen D. Malony*//*
*//*    - HPDC Achievement Award winner: Ewa Deelman*/

HPDC?15, the 24th International ACM Symposium on High Performance 
Parallel and Distributed Computing, will be held in Portland, Oregon, 
June 15-19, 2015. HPDC?15 will be part of FCRC?15, ACM?s Federated 
Computing Research Conference, hosting no less than 13 top-level 
conferences and symposia, along with an exciting program of joint 
keynote lectures, opening on the evening of June 14th with the A.M. 
Turing Award Lecture, delivered by the 2014 award winner, Michael 
Stonebreaker.

HPDC is hosting a total of seven workshops on June 15 and 16. The main 
HPDC conference will run from June 17 to 19, featuring 30 presentations 
of papers and short papers, two keynotes, and a poster session. This 
year?s HPDC keynote presentations will be given by Allen D. Malony and 
this year?s HPDC Achievement Award winner, Ewa Deelman.

Please see http://www.hpdc.org/2015/ for details. Registration (via 
FCRC) is open now. Deadline for discounted, early registration is May 
18, 2015. Travel grants for students attending a US institution are 
available, sponsored by NSF.

*Call for posters:**
*
HPDC'15 will feature a poster session that will provide the right 
environment for lively and informal discussions on various high 
performance parallel and distributed computing topics. There will be a 
"poster session and reception" on Wednesday, 6-8 pm. Posters can remain 
on display during the three days of the HPDC main conference (Wednesday 
to Friday). We invite all potential authors to submit their contribution 
to this poster session in the form of a two-page PDF abstract (we 
recommend using the ACM Proceedings style, and fonts not smaller than 10 
point). Posters may be accompanied by practical demonstrations. 
Abstracts must be submitted by May 8th 2015 through 
https://www.easychair.org/conferences/?conf=hpdc15posters. For any 
questions about the submission, selection, and presentation of the 
accepted posters, please contact the Posters Chair - Ana-Maria Oprescu 
<amo at cs.vu.nl>, Vrije Universiteit Amsterdam, The Netherlands.




-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150506/ea35215d/attachment.html>

From yadunand at uchicago.edu  Fri May  8 11:06:23 2015
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Fri, 08 May 2015 11:06:23 -0500
Subject: [Swift-devel] Updating swift-lang for 0.96.0
Message-ID: <554CDEFF.8010808@uchicago.edu>

Hi All,

I'm updating swift-lang.org for the 0.96.0 release, and should be done 
in an hour.
I would really appreciate it if folks could try links and go over the 
website text to
check if everything is in order.

Thanks,
Yadu


From ketan at mcs.anl.gov  Fri May  8 12:40:01 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 12:40:01 -0500
Subject: [Swift-devel] Updating swift-lang for 0.96.0
In-Reply-To: <554CDEFF.8010808@uchicago.edu>
References: <554CDEFF.8010808@uchicago.edu>
Message-ID: <CAMUuvip3X7iVUcDfOSyL6sjSmNWsyjw9P9NVuQ5K5dV3w7yF2Q@mail.gmail.com>

Looks good to me.

Comments:

-- in the release number we had two parts x.xx (0.95) whereas in this one
it is x.xx.x (0.96.0). Intentional?
-- Release notes points to 404 not found (may be disable link for now?)
-- Configuration file format link points to your home www area, may be
point to github conf schema link (
https://github.com/swift-lang/swift-k/blob/master/resources/swift.conf.schema)
or swift-lang.org area.



On Fri, May 8, 2015 at 11:06 AM, Yadu Nand Babuji <yadunand at uchicago.edu>
wrote:

> Hi All,
>
> I'm updating swift-lang.org for the 0.96.0 release, and should be done
> in an hour.
> I would really appreciate it if folks could try links and go over the
> website text to
> check if everything is in order.
>
> Thanks,
> Yadu
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/faac0609/attachment.html>

From ketan at mcs.anl.gov  Fri May  8 13:00:37 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 13:00:37 -0500
Subject: [Swift-devel] qsub env
Message-ID: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>

Hi,

Question about the cobalt provider with 0.96: how to set an environment so
that it ends up on the qsub line:

qsub --env MY_ENV=true ...

I tried to add the env.MY_ENV line under site.cluster
and site.cluster.execution.options but to no avail.

>From logs of my current runs, I see that the qsub line contains one
environment variable:

qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...

but I do not see it being specified anywhere in my conf file.

Thanks for any suggestions.

--
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/dd6eeb9b/attachment.html>

From hategan at mcs.anl.gov  Fri May  8 13:18:29 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 8 May 2015 11:18:29 -0700
Subject: [Swift-devel] qsub env
In-Reply-To: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
Message-ID: <1431109109.6021.1.camel@echo>

Are you trying to pass an environment variable to an app or to the
worker?

Mihael

On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> Hi,
> 
> Question about the cobalt provider with 0.96: how to set an environment so
> that it ends up on the qsub line:
> 
> qsub --env MY_ENV=true ...
> 
> I tried to add the env.MY_ENV line under site.cluster
> and site.cluster.execution.options but to no avail.
> 
> From logs of my current runs, I see that the qsub line contains one
> environment variable:
> 
> qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...
> 
> but I do not see it being specified anywhere in my conf file.
> 
> Thanks for any suggestions.
> 
> --
> Ketan
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Fri May  8 13:34:27 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 13:34:27 -0500
Subject: [Swift-devel] qsub env
In-Reply-To: <1431109109.6021.1.camel@echo>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
Message-ID: <CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>

Hi Mihael,

I am trying to pass an environment variable at the qsub level.

In the Swift source, CobalExecutor.java, I find the following code as part
of qsub commandline builder:

Collection<String> names = getSpec().getEnvironmentVariableNames();
        if (names != null && names.size() > 0) {
            result.add("--env");
            StringBuffer sb = new StringBuffer();
            Iterator<String> i = names.iterator();
            while (i.hasNext()) {
                String name = i.next();
                sb.append(name);
                sb.append('=');
                sb.append(quote(getSpec().getEnvironmentVariable(name)));
                if (i.hasNext()) {
                    sb.append(':');
                }
            }
            result.add(sb.toString());

I am wondering how to add my environment variable to this list.

--
Ketan


On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Are you trying to pass an environment variable to an app or to the
> worker?
>
> Mihael
>
> On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > Hi,
> >
> > Question about the cobalt provider with 0.96: how to set an environment
> so
> > that it ends up on the qsub line:
> >
> > qsub --env MY_ENV=true ...
> >
> > I tried to add the env.MY_ENV line under site.cluster
> > and site.cluster.execution.options but to no avail.
> >
> > From logs of my current runs, I see that the qsub line contains one
> > environment variable:
> >
> > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...
> >
> > but I do not see it being specified anywhere in my conf file.
> >
> > Thanks for any suggestions.
> >
> > --
> > Ketan
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/07ad2ecb/attachment.html>

From hategan at mcs.anl.gov  Fri May  8 14:02:26 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 8 May 2015 12:02:26 -0700
Subject: [Swift-devel] qsub env
In-Reply-To: <CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
Message-ID: <1431111746.6660.1.camel@echo>

Let me rephrase the question. What entity uses this environment
variable?

Mihael

On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> Hi Mihael,
> 
> I am trying to pass an environment variable at the qsub level.
> 
> In the Swift source, CobalExecutor.java, I find the following code as part
> of qsub commandline builder:
> 
> Collection<String> names = getSpec().getEnvironmentVariableNames();
>         if (names != null && names.size() > 0) {
>             result.add("--env");
>             StringBuffer sb = new StringBuffer();
>             Iterator<String> i = names.iterator();
>             while (i.hasNext()) {
>                 String name = i.next();
>                 sb.append(name);
>                 sb.append('=');
>                 sb.append(quote(getSpec().getEnvironmentVariable(name)));
>                 if (i.hasNext()) {
>                     sb.append(':');
>                 }
>             }
>             result.add(sb.toString());
> 
> I am wondering how to add my environment variable to this list.
> 
> --
> Ketan
> 
> 
> On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > Are you trying to pass an environment variable to an app or to the
> > worker?
> >
> > Mihael
> >
> > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > Hi,
> > >
> > > Question about the cobalt provider with 0.96: how to set an environment
> > so
> > > that it ends up on the qsub line:
> > >
> > > qsub --env MY_ENV=true ...
> > >
> > > I tried to add the env.MY_ENV line under site.cluster
> > > and site.cluster.execution.options but to no avail.
> > >
> > > From logs of my current runs, I see that the qsub line contains one
> > > environment variable:
> > >
> > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...
> > >
> > > but I do not see it being specified anywhere in my conf file.
> > >
> > > Thanks for any suggestions.
> > >
> > > --
> > > Ketan
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >




From ketan at mcs.anl.gov  Fri May  8 14:04:28 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 14:04:28 -0500
Subject: [Swift-devel] qsub env
In-Reply-To: <1431111746.6660.1.camel@echo>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
Message-ID: <CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>

Cobalt uses the environment variable in the way it allocates compute
blocks: bootable vs non-bootable. --Ketan

On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> Let me rephrase the question. What entity uses this environment
> variable?
>
> Mihael
>
> On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > Hi Mihael,
> >
> > I am trying to pass an environment variable at the qsub level.
> >
> > In the Swift source, CobalExecutor.java, I find the following code as
> part
> > of qsub commandline builder:
> >
> > Collection<String> names = getSpec().getEnvironmentVariableNames();
> >         if (names != null && names.size() > 0) {
> >             result.add("--env");
> >             StringBuffer sb = new StringBuffer();
> >             Iterator<String> i = names.iterator();
> >             while (i.hasNext()) {
> >                 String name = i.next();
> >                 sb.append(name);
> >                 sb.append('=');
> >                 sb.append(quote(getSpec().getEnvironmentVariable(name)));
> >                 if (i.hasNext()) {
> >                     sb.append(':');
> >                 }
> >             }
> >             result.add(sb.toString());
> >
> > I am wondering how to add my environment variable to this list.
> >
> > --
> > Ketan
> >
> >
> > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> >
> > > Are you trying to pass an environment variable to an app or to the
> > > worker?
> > >
> > > Mihael
> > >
> > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > Hi,
> > > >
> > > > Question about the cobalt provider with 0.96: how to set an
> environment
> > > so
> > > > that it ends up on the qsub line:
> > > >
> > > > qsub --env MY_ENV=true ...
> > > >
> > > > I tried to add the env.MY_ENV line under site.cluster
> > > > and site.cluster.execution.options but to no avail.
> > > >
> > > > From logs of my current runs, I see that the qsub line contains one
> > > > environment variable:
> > > >
> > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...
> > > >
> > > > but I do not see it being specified anywhere in my conf file.
> > > >
> > > > Thanks for any suggestions.
> > > >
> > > > --
> > > > Ketan
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/c7e3ad3b/attachment.html>

From hategan at mcs.anl.gov  Fri May  8 14:22:32 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 8 May 2015 12:22:32 -0700
Subject: [Swift-devel] qsub env
In-Reply-To: <CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
	<CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
Message-ID: <1431112952.6932.4.camel@echo>

There is no user-customizable way to pass environment variables to the
block job through coasters, mostly because worker.pl is the only entity
that could use them.

So this should be a cobalt provider option. The way you would do it is
similar to how, for example, PBS handles custom options (see, for
example pbs.resource_list).

If you give me the specifics, I could do it.

Mihael


On Fri, 2015-05-08 at 14:04 -0500, Ketan Maheshwari wrote:
> Cobalt uses the environment variable in the way it allocates compute
> blocks: bootable vs non-bootable. --Ketan
> 
> On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > Let me rephrase the question. What entity uses this environment
> > variable?
> >
> > Mihael
> >
> > On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > > Hi Mihael,
> > >
> > > I am trying to pass an environment variable at the qsub level.
> > >
> > > In the Swift source, CobalExecutor.java, I find the following code as
> > part
> > > of qsub commandline builder:
> > >
> > > Collection<String> names = getSpec().getEnvironmentVariableNames();
> > >         if (names != null && names.size() > 0) {
> > >             result.add("--env");
> > >             StringBuffer sb = new StringBuffer();
> > >             Iterator<String> i = names.iterator();
> > >             while (i.hasNext()) {
> > >                 String name = i.next();
> > >                 sb.append(name);
> > >                 sb.append('=');
> > >                 sb.append(quote(getSpec().getEnvironmentVariable(name)));
> > >                 if (i.hasNext()) {
> > >                     sb.append(':');
> > >                 }
> > >             }
> > >             result.add(sb.toString());
> > >
> > > I am wondering how to add my environment variable to this list.
> > >
> > > --
> > > Ketan
> > >
> > >
> > > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> > >
> > > > Are you trying to pass an environment variable to an app or to the
> > > > worker?
> > > >
> > > > Mihael
> > > >
> > > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > > Hi,
> > > > >
> > > > > Question about the cobalt provider with 0.96: how to set an
> > environment
> > > > so
> > > > > that it ends up on the qsub line:
> > > > >
> > > > > qsub --env MY_ENV=true ...
> > > > >
> > > > > I tried to add the env.MY_ENV line under site.cluster
> > > > > and site.cluster.execution.options but to no avail.
> > > > >
> > > > > From logs of my current runs, I see that the qsub line contains one
> > > > > environment variable:
> > > > >
> > > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount 32 ...
> > > > >
> > > > > but I do not see it being specified anywhere in my conf file.
> > > > >
> > > > > Thanks for any suggestions.
> > > > >
> > > > > --
> > > > > Ketan
> > > > > _______________________________________________
> > > > > Swift-devel mailing list
> > > > > Swift-devel at ci.uchicago.edu
> > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >




From ketan at mcs.anl.gov  Fri May  8 14:41:45 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 14:41:45 -0500
Subject: [Swift-devel] qsub env
In-Reply-To: <1431112952.6932.4.camel@echo>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
	<CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
	<1431112952.6932.4.camel@echo>
Message-ID: <CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>

In terms of cobalt provider option, it would be the switch
"--disable_preboot". The corresponding environment variable is "--env
RUNJOB_DISABLE_PREBOOT=true". --Ketan

On Fri, May 8, 2015 at 2:22 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> There is no user-customizable way to pass environment variables to the
> block job through coasters, mostly because worker.pl is the only entity
> that could use them.
>
> So this should be a cobalt provider option. The way you would do it is
> similar to how, for example, PBS handles custom options (see, for
> example pbs.resource_list).
>
> If you give me the specifics, I could do it.
>
> Mihael
>
>
> On Fri, 2015-05-08 at 14:04 -0500, Ketan Maheshwari wrote:
> > Cobalt uses the environment variable in the way it allocates compute
> > blocks: bootable vs non-bootable. --Ketan
> >
> > On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> >
> > > Let me rephrase the question. What entity uses this environment
> > > variable?
> > >
> > > Mihael
> > >
> > > On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > > > Hi Mihael,
> > > >
> > > > I am trying to pass an environment variable at the qsub level.
> > > >
> > > > In the Swift source, CobalExecutor.java, I find the following code as
> > > part
> > > > of qsub commandline builder:
> > > >
> > > > Collection<String> names = getSpec().getEnvironmentVariableNames();
> > > >         if (names != null && names.size() > 0) {
> > > >             result.add("--env");
> > > >             StringBuffer sb = new StringBuffer();
> > > >             Iterator<String> i = names.iterator();
> > > >             while (i.hasNext()) {
> > > >                 String name = i.next();
> > > >                 sb.append(name);
> > > >                 sb.append('=');
> > > >
>  sb.append(quote(getSpec().getEnvironmentVariable(name)));
> > > >                 if (i.hasNext()) {
> > > >                     sb.append(':');
> > > >                 }
> > > >             }
> > > >             result.add(sb.toString());
> > > >
> > > > I am wondering how to add my environment variable to this list.
> > > >
> > > > --
> > > > Ketan
> > > >
> > > >
> > > > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > > wrote:
> > > >
> > > > > Are you trying to pass an environment variable to an app or to the
> > > > > worker?
> > > > >
> > > > > Mihael
> > > > >
> > > > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Question about the cobalt provider with 0.96: how to set an
> > > environment
> > > > > so
> > > > > > that it ends up on the qsub line:
> > > > > >
> > > > > > qsub --env MY_ENV=true ...
> > > > > >
> > > > > > I tried to add the env.MY_ENV line under site.cluster
> > > > > > and site.cluster.execution.options but to no avail.
> > > > > >
> > > > > > From logs of my current runs, I see that the qsub line contains
> one
> > > > > > environment variable:
> > > > > >
> > > > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount
> 32 ...
> > > > > >
> > > > > > but I do not see it being specified anywhere in my conf file.
> > > > > >
> > > > > > Thanks for any suggestions.
> > > > > >
> > > > > > --
> > > > > > Ketan
> > > > > > _______________________________________________
> > > > > > Swift-devel mailing list
> > > > > > Swift-devel at ci.uchicago.edu
> > > > > >
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Swift-devel mailing list
> > > > > Swift-devel at ci.uchicago.edu
> > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/e6f4789b/attachment.html>

From hategan at mcs.anl.gov  Fri May  8 14:46:34 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 8 May 2015 12:46:34 -0700
Subject: [Swift-devel] qsub env
In-Reply-To: <CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
	<CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
	<1431112952.6932.4.camel@echo>
	<CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>
Message-ID: <1431114394.7508.0.camel@echo>

"true" and "false" being the valid values?

On Fri, 2015-05-08 at 14:41 -0500, Ketan Maheshwari wrote:
> In terms of cobalt provider option, it would be the switch
> "--disable_preboot". The corresponding environment variable is "--env
> RUNJOB_DISABLE_PREBOOT=true". --Ketan
> 
> On Fri, May 8, 2015 at 2:22 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > There is no user-customizable way to pass environment variables to the
> > block job through coasters, mostly because worker.pl is the only entity
> > that could use them.
> >
> > So this should be a cobalt provider option. The way you would do it is
> > similar to how, for example, PBS handles custom options (see, for
> > example pbs.resource_list).
> >
> > If you give me the specifics, I could do it.
> >
> > Mihael
> >
> >
> > On Fri, 2015-05-08 at 14:04 -0500, Ketan Maheshwari wrote:
> > > Cobalt uses the environment variable in the way it allocates compute
> > > blocks: bootable vs non-bootable. --Ketan
> > >
> > > On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> > >
> > > > Let me rephrase the question. What entity uses this environment
> > > > variable?
> > > >
> > > > Mihael
> > > >
> > > > On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > > > > Hi Mihael,
> > > > >
> > > > > I am trying to pass an environment variable at the qsub level.
> > > > >
> > > > > In the Swift source, CobalExecutor.java, I find the following code as
> > > > part
> > > > > of qsub commandline builder:
> > > > >
> > > > > Collection<String> names = getSpec().getEnvironmentVariableNames();
> > > > >         if (names != null && names.size() > 0) {
> > > > >             result.add("--env");
> > > > >             StringBuffer sb = new StringBuffer();
> > > > >             Iterator<String> i = names.iterator();
> > > > >             while (i.hasNext()) {
> > > > >                 String name = i.next();
> > > > >                 sb.append(name);
> > > > >                 sb.append('=');
> > > > >
> >  sb.append(quote(getSpec().getEnvironmentVariable(name)));
> > > > >                 if (i.hasNext()) {
> > > > >                     sb.append(':');
> > > > >                 }
> > > > >             }
> > > > >             result.add(sb.toString());
> > > > >
> > > > > I am wondering how to add my environment variable to this list.
> > > > >
> > > > > --
> > > > > Ketan
> > > > >
> > > > >
> > > > > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > > > wrote:
> > > > >
> > > > > > Are you trying to pass an environment variable to an app or to the
> > > > > > worker?
> > > > > >
> > > > > > Mihael
> > > > > >
> > > > > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Question about the cobalt provider with 0.96: how to set an
> > > > environment
> > > > > > so
> > > > > > > that it ends up on the qsub line:
> > > > > > >
> > > > > > > qsub --env MY_ENV=true ...
> > > > > > >
> > > > > > > I tried to add the env.MY_ENV line under site.cluster
> > > > > > > and site.cluster.execution.options but to no avail.
> > > > > > >
> > > > > > > From logs of my current runs, I see that the qsub line contains
> > one
> > > > > > > environment variable:
> > > > > > >
> > > > > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount
> > 32 ...
> > > > > > >
> > > > > > > but I do not see it being specified anywhere in my conf file.
> > > > > > >
> > > > > > > Thanks for any suggestions.
> > > > > > >
> > > > > > > --
> > > > > > > Ketan
> > > > > > > _______________________________________________
> > > > > > > Swift-devel mailing list
> > > > > > > Swift-devel at ci.uchicago.edu
> > > > > > >
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Swift-devel mailing list
> > > > > > Swift-devel at ci.uchicago.edu
> > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >




From ketan at mcs.anl.gov  Fri May  8 14:51:51 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 8 May 2015 14:51:51 -0500
Subject: [Swift-devel] qsub env
In-Reply-To: <1431114394.7508.0.camel@echo>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
	<CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
	<1431112952.6932.4.camel@echo>
	<CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>
	<1431114394.7508.0.camel@echo>
Message-ID: <CAMUuviq9jvuwOq9DdxAFRA3c_BW3BELjOtXQHftYzB-x0Pw8Pg@mail.gmail.com>

No, it is an "implicit" variable. Its presence means true, otherwise false.

On Fri, May 8, 2015 at 2:46 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:

> "true" and "false" being the valid values?
>
> On Fri, 2015-05-08 at 14:41 -0500, Ketan Maheshwari wrote:
> > In terms of cobalt provider option, it would be the switch
> > "--disable_preboot". The corresponding environment variable is "--env
> > RUNJOB_DISABLE_PREBOOT=true". --Ketan
> >
> > On Fri, May 8, 2015 at 2:22 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
> >
> > > There is no user-customizable way to pass environment variables to the
> > > block job through coasters, mostly because worker.pl is the only
> entity
> > > that could use them.
> > >
> > > So this should be a cobalt provider option. The way you would do it is
> > > similar to how, for example, PBS handles custom options (see, for
> > > example pbs.resource_list).
> > >
> > > If you give me the specifics, I could do it.
> > >
> > > Mihael
> > >
> > >
> > > On Fri, 2015-05-08 at 14:04 -0500, Ketan Maheshwari wrote:
> > > > Cobalt uses the environment variable in the way it allocates compute
> > > > blocks: bootable vs non-bootable. --Ketan
> > > >
> > > > On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > > wrote:
> > > >
> > > > > Let me rephrase the question. What entity uses this environment
> > > > > variable?
> > > > >
> > > > > Mihael
> > > > >
> > > > > On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > > > > > Hi Mihael,
> > > > > >
> > > > > > I am trying to pass an environment variable at the qsub level.
> > > > > >
> > > > > > In the Swift source, CobalExecutor.java, I find the following
> code as
> > > > > part
> > > > > > of qsub commandline builder:
> > > > > >
> > > > > > Collection<String> names =
> getSpec().getEnvironmentVariableNames();
> > > > > >         if (names != null && names.size() > 0) {
> > > > > >             result.add("--env");
> > > > > >             StringBuffer sb = new StringBuffer();
> > > > > >             Iterator<String> i = names.iterator();
> > > > > >             while (i.hasNext()) {
> > > > > >                 String name = i.next();
> > > > > >                 sb.append(name);
> > > > > >                 sb.append('=');
> > > > > >
> > >  sb.append(quote(getSpec().getEnvironmentVariable(name)));
> > > > > >                 if (i.hasNext()) {
> > > > > >                     sb.append(':');
> > > > > >                 }
> > > > > >             }
> > > > > >             result.add(sb.toString());
> > > > > >
> > > > > > I am wondering how to add my environment variable to this list.
> > > > > >
> > > > > > --
> > > > > > Ketan
> > > > > >
> > > > > >
> > > > > > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <
> hategan at mcs.anl.gov>
> > > > > wrote:
> > > > > >
> > > > > > > Are you trying to pass an environment variable to an app or to
> the
> > > > > > > worker?
> > > > > > >
> > > > > > > Mihael
> > > > > > >
> > > > > > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Question about the cobalt provider with 0.96: how to set an
> > > > > environment
> > > > > > > so
> > > > > > > > that it ends up on the qsub line:
> > > > > > > >
> > > > > > > > qsub --env MY_ENV=true ...
> > > > > > > >
> > > > > > > > I tried to add the env.MY_ENV line under site.cluster
> > > > > > > > and site.cluster.execution.options but to no avail.
> > > > > > > >
> > > > > > > > From logs of my current runs, I see that the qsub line
> contains
> > > one
> > > > > > > > environment variable:
> > > > > > > >
> > > > > > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script
> --proccount
> > > 32 ...
> > > > > > > >
> > > > > > > > but I do not see it being specified anywhere in my conf file.
> > > > > > > >
> > > > > > > > Thanks for any suggestions.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Ketan
> > > > > > > > _______________________________________________
> > > > > > > > Swift-devel mailing list
> > > > > > > > Swift-devel at ci.uchicago.edu
> > > > > > > >
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Swift-devel mailing list
> > > > > > > Swift-devel at ci.uchicago.edu
> > > > > > >
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Swift-devel mailing list
> > > > > Swift-devel at ci.uchicago.edu
> > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > >
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150508/daf57553/attachment.html>

From hategan at mcs.anl.gov  Fri May  8 14:52:17 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 8 May 2015 12:52:17 -0700
Subject: [Swift-devel] qsub env
In-Reply-To: <CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>
References: <CAMUuvir-qEmhxMpAssZoLWLfAa80GUeD41HOA4Q9AySekXY1pg@mail.gmail.com>
	<1431109109.6021.1.camel@echo>
	<CAMUuvipRYSWz=vO5NuMQu4ffzOZN+itHoWgHWduW=x3EJXSJBQ@mail.gmail.com>
	<1431111746.6660.1.camel@echo>
	<CAMUuvirG4AxBT1JvsXavLHNxvvFLCRrHKGJBm49OJ2USXP0b0Q@mail.gmail.com>
	<1431112952.6932.4.camel@echo>
	<CAMUuvipiH-3hwKcYch8iDz6PinGC-44ws6N7FhS6O9v-mqTkCg@mail.gmail.com>
Message-ID: <1431114737.7508.3.camel@echo>

Done:

options.jobOptions.cobalt.disablePreboot: <boolean>

It was a one liner in Cobalt executor. For reference, if you need to do
something similar:
https://github.com/swift-lang/swift-k/commit/4177dc0a6f7597c05924df10f0b617df535f5c5a

Let me know if it doesn't work properly.

Mihael

On Fri, 2015-05-08 at 14:41 -0500, Ketan Maheshwari wrote:
> In terms of cobalt provider option, it would be the switch
> "--disable_preboot". The corresponding environment variable is "--env
> RUNJOB_DISABLE_PREBOOT=true". --Ketan
> 
> On Fri, May 8, 2015 at 2:22 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> > There is no user-customizable way to pass environment variables to the
> > block job through coasters, mostly because worker.pl is the only entity
> > that could use them.
> >
> > So this should be a cobalt provider option. The way you would do it is
> > similar to how, for example, PBS handles custom options (see, for
> > example pbs.resource_list).
> >
> > If you give me the specifics, I could do it.
> >
> > Mihael
> >
> >
> > On Fri, 2015-05-08 at 14:04 -0500, Ketan Maheshwari wrote:
> > > Cobalt uses the environment variable in the way it allocates compute
> > > blocks: bootable vs non-bootable. --Ketan
> > >
> > > On Fri, May 8, 2015 at 2:02 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > wrote:
> > >
> > > > Let me rephrase the question. What entity uses this environment
> > > > variable?
> > > >
> > > > Mihael
> > > >
> > > > On Fri, 2015-05-08 at 13:34 -0500, Ketan Maheshwari wrote:
> > > > > Hi Mihael,
> > > > >
> > > > > I am trying to pass an environment variable at the qsub level.
> > > > >
> > > > > In the Swift source, CobalExecutor.java, I find the following code as
> > > > part
> > > > > of qsub commandline builder:
> > > > >
> > > > > Collection<String> names = getSpec().getEnvironmentVariableNames();
> > > > >         if (names != null && names.size() > 0) {
> > > > >             result.add("--env");
> > > > >             StringBuffer sb = new StringBuffer();
> > > > >             Iterator<String> i = names.iterator();
> > > > >             while (i.hasNext()) {
> > > > >                 String name = i.next();
> > > > >                 sb.append(name);
> > > > >                 sb.append('=');
> > > > >
> >  sb.append(quote(getSpec().getEnvironmentVariable(name)));
> > > > >                 if (i.hasNext()) {
> > > > >                     sb.append(':');
> > > > >                 }
> > > > >             }
> > > > >             result.add(sb.toString());
> > > > >
> > > > > I am wondering how to add my environment variable to this list.
> > > > >
> > > > > --
> > > > > Ketan
> > > > >
> > > > >
> > > > > On Fri, May 8, 2015 at 1:18 PM, Mihael Hategan <hategan at mcs.anl.gov>
> > > > wrote:
> > > > >
> > > > > > Are you trying to pass an environment variable to an app or to the
> > > > > > worker?
> > > > > >
> > > > > > Mihael
> > > > > >
> > > > > > On Fri, 2015-05-08 at 13:00 -0500, Ketan Maheshwari wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Question about the cobalt provider with 0.96: how to set an
> > > > environment
> > > > > > so
> > > > > > > that it ends up on the qsub line:
> > > > > > >
> > > > > > > qsub --env MY_ENV=true ...
> > > > > > >
> > > > > > > I tried to add the env.MY_ENV line under site.cluster
> > > > > > > and site.cluster.execution.options but to no avail.
> > > > > > >
> > > > > > > From logs of my current runs, I see that the qsub line contains
> > one
> > > > > > > environment variable:
> > > > > > >
> > > > > > > qsub --env WORKER_LOGGING_LEVEL=NONE --mode script --proccount
> > 32 ...
> > > > > > >
> > > > > > > but I do not see it being specified anywhere in my conf file.
> > > > > > >
> > > > > > > Thanks for any suggestions.
> > > > > > >
> > > > > > > --
> > > > > > > Ketan
> > > > > > > _______________________________________________
> > > > > > > Swift-devel mailing list
> > > > > > > Swift-devel at ci.uchicago.edu
> > > > > > >
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Swift-devel mailing list
> > > > > > Swift-devel at ci.uchicago.edu
> > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Swift-devel mailing list
> > > > Swift-devel at ci.uchicago.edu
> > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > > >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >




From ketan at mcs.anl.gov  Fri May 15 11:18:40 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 15 May 2015 11:18:40 -0500
Subject: [Swift-devel] appending the app name with 'app-' results in parsing
	error
Message-ID: <CAMUuviptfHH+H3T0Hq5Qm8v7enf6+bkvW-Mtzr80MMH7yX90aA@mail.gmail.com>

Hi,

For example:

type file;

app (file o) app-cat (file i) {
  cat @i stdout=@o;
}
file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;

string d="data.txt";

foreach j in [1:10] {
 file data<single_file_mapper; file=d>;
 out[j] = app-cat(data);
}

Gives:

$ swift -config local.conf catsn.swift
Swift trunk git-rev: 86de22e2841518dbf337a0e0da56ad1dd9dd70f8 heads/master 6343
RunID: run003
Could not compile SwiftScript source: line 3:14: expecting an
identifier, found 'app'

It works with 'app_' (ie. replaceing hyphen with underscore).

Bug?

--
Ketan


From hategan at mcs.anl.gov  Fri May 15 11:24:51 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 15 May 2015 09:24:51 -0700
Subject: [Swift-devel] appending the app name with 'app-' results in
 parsing error
In-Reply-To: <CAMUuviptfHH+H3T0Hq5Qm8v7enf6+bkvW-Mtzr80MMH7yX90aA@mail.gmail.com>
References: <CAMUuviptfHH+H3T0Hq5Qm8v7enf6+bkvW-Mtzr80MMH7yX90aA@mail.gmail.com>
Message-ID: <1431707091.28435.2.camel@echo3>

Not bug. "-" is an operator and if it was allowed in identifiers, a-b
would be ambiguous. You could try the same thing in Java, C, or any
other language that has the basic math operators.

Mihael

On Fri, 2015-05-15 at 11:18 -0500, Ketan Maheshwari wrote:
> Hi,
> 
> For example:
> 
> type file;
> 
> app (file o) app-cat (file i) {
>   cat @i stdout=@o;
> }
> file out[]<simple_mapper; location="outdir", prefix="f.",suffix=".out">;
> 
> string d="data.txt";
> 
> foreach j in [1:10] {
>  file data<single_file_mapper; file=d>;
>  out[j] = app-cat(data);
> }
> 
> Gives:
> 
> $ swift -config local.conf catsn.swift
> Swift trunk git-rev: 86de22e2841518dbf337a0e0da56ad1dd9dd70f8 heads/master 6343
> RunID: run003
> Could not compile SwiftScript source: line 3:14: expecting an
> identifier, found 'app'
> 
> It works with 'app_' (ie. replaceing hyphen with underscore).
> 
> Bug?
> 
> --
> Ketan
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Fri May 15 11:56:08 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Fri, 15 May 2015 11:56:08 -0500
Subject: [Swift-devel] exec not in path despite providing absolute path
Message-ID: <CAMUuvioWtK2au3VB26QzeZkdagfZWeT3xgqWURTVFNhDep1tXw@mail.gmail.com>

Hi,

I am providing absolute path to an executable but Swift is complaining
that it is not in path. Here is the error message:

$ swift -config local.conf nek.swift
Swift trunk git-rev: 86de22e2841518dbf337a0e0da56ad1dd9dd70f8 heads/master 6343
RunID: run004
Progress: Fri, 15 May 2015 11:51:55-0500
Execution failed:
Exception in genrun:
    Arguments: [-d, LST.json, -u, LST_f90.tusr, --makenek,
~/maxhutch/nek/makenek, RTI_LST]
    Host: local
    Directory: nek-run004/jobs/g/genrun-aj3y9t8m
exception @ swift-int.k, line: 467
swiftStaging:execute2 @ swift.k, line: 139
executeSelect @ swift.k, line: 207
Caused by: Cannot find executable genrun on site system path
throw @ swift-int.k, line: 78
checkJobStatus @ swift-int.k, line: 406

k:assign @ swift.k, line: 174
swift:execute @ nek, line: 46
app_genrun @ nek, line: 46


Here is the config:

$ cat local.conf
site.localhost {
    execution {
        type: "local"
        URL: "localhost"
    }
    filesystem {
        type: "local"
        URL: "localhost"
    }
    workDirectory: "/tmp/swift.workdir"
    app.cat {
        executable: "/bin/cat"
    }
    app.genrun {
        executable: "/home/ketan/maxhutch/nek-tools/genrun/genrun.py"
    }
}

lazyErrors: false
executionRetries: 0
keepSiteDir: true
providerStagingPinSwiftFiles: false
alwaysTransferWrapperLog: true

The log dir is attached.

Any suggestions?

Thanks,
--
Ketan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run004.tgz
Type: application/x-gzip
Size: 5772 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150515/c0bd9816/attachment.bin>

From hategan at mcs.anl.gov  Sat May 16 16:04:41 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 16 May 2015 14:04:41 -0700
Subject: [Swift-devel] user guide work
Message-ID: <1431810281.23991.3.camel@echo3>

Hi,

Mike pointed out that I should probably post a link to what I'm trying
to do with the user guide just in case things are going awfully wrong.
So here it is:

http://www.mcs.anl.gov/~hategan/ug/ug/ug.html

Mike mention K&R a number of times and I think I like that general idea.
The above, so far, is a mix between K&R Chapter 2 and Appendix A.

The main idea is to keep things concise by separating parallelism
behaviour from what a program actually does. In that sense, one would
first describe the language, and then go into the details of how a
particular implementation goes about running programs.

I'm not particularly opposed to starting with a brief tutorial in the
spirit of K&R Chapter 1.

Mihael




From hategan at mcs.anl.gov  Sat May 16 16:11:34 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Sat, 16 May 2015 14:11:34 -0700
Subject: [Swift-devel] exec not in path despite providing absolute path
In-Reply-To: <CAMUuvioWtK2au3VB26QzeZkdagfZWeT3xgqWURTVFNhDep1tXw@mail.gmail.com>
References: <CAMUuvioWtK2au3VB26QzeZkdagfZWeT3xgqWURTVFNhDep1tXw@mail.gmail.com>
Message-ID: <1431810694.23991.6.camel@echo3>

The magic of hidden includes.

You are not actually using the site you defined. You probably want to
say 

sites: [localhost]

at some point in that config file.

What's happening is that "site: [local]" from etc/swift.conf is used,
and that does not contain your app.

Mihael

On Fri, 2015-05-15 at 11:56 -0500, Ketan Maheshwari wrote:
> Hi,
> 
> I am providing absolute path to an executable but Swift is complaining
> that it is not in path. Here is the error message:
> 
> $ swift -config local.conf nek.swift
> Swift trunk git-rev: 86de22e2841518dbf337a0e0da56ad1dd9dd70f8 heads/master 6343
> RunID: run004
> Progress: Fri, 15 May 2015 11:51:55-0500
> Execution failed:
> Exception in genrun:
>     Arguments: [-d, LST.json, -u, LST_f90.tusr, --makenek,
> ~/maxhutch/nek/makenek, RTI_LST]
>     Host: local
>     Directory: nek-run004/jobs/g/genrun-aj3y9t8m
> exception @ swift-int.k, line: 467
> swiftStaging:execute2 @ swift.k, line: 139
> executeSelect @ swift.k, line: 207
> Caused by: Cannot find executable genrun on site system path
> throw @ swift-int.k, line: 78
> checkJobStatus @ swift-int.k, line: 406
> 
> k:assign @ swift.k, line: 174
> swift:execute @ nek, line: 46
> app_genrun @ nek, line: 46
> 
> 
> Here is the config:
> 
> $ cat local.conf
> site.localhost {
>     execution {
>         type: "local"
>         URL: "localhost"
>     }
>     filesystem {
>         type: "local"
>         URL: "localhost"
>     }
>     workDirectory: "/tmp/swift.workdir"
>     app.cat {
>         executable: "/bin/cat"
>     }
>     app.genrun {
>         executable: "/home/ketan/maxhutch/nek-tools/genrun/genrun.py"
>     }
> }
> 
> lazyErrors: false
> executionRetries: 0
> keepSiteDir: true
> providerStagingPinSwiftFiles: false
> alwaysTransferWrapperLog: true
> 
> The log dir is attached.
> 
> Any suggestions?
> 
> Thanks,
> --
> Ketan
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Mon May 18 13:27:26 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 13:27:26 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <1431810281.23991.3.camel@echo3>
References: <1431810281.23991.3.camel@echo3>
Message-ID: <CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>

I think Swift is a special purpose language where ~20% of features are
used for ~80% of cases. Consequently, providing a K&R style language
manual might not be the most valuable thing for a user to read. It
also runs a risk of coming across as a general purpose language which
may mislead readers.

I think, a document structure where the important features (app and
builtin functions, variables and mappers (with examples)) are
highlighted in the early parts and details such as variable scope,
expressions, operators etc. are described in the middle and concluded
with sites specific details might serve well.

--
Ketan

On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> Hi,
>
> Mike pointed out that I should probably post a link to what I'm trying
> to do with the user guide just in case things are going awfully wrong.
> So here it is:
>
> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>
> Mike mention K&R a number of times and I think I like that general idea.
> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>
> The main idea is to keep things concise by separating parallelism
> behaviour from what a program actually does. In that sense, one would
> first describe the language, and then go into the details of how a
> particular implementation goes about running programs.
>
> I'm not particularly opposed to starting with a brief tutorial in the
> spirit of K&R Chapter 1.
>
> Mihael
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Mon May 18 14:20:53 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 May 2015 12:20:53 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
Message-ID: <1431976853.12263.13.camel@echo3>

On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
> I think Swift is a special purpose language where ~20% of features are
> used for ~80% of cases. Consequently, providing a K&R style language
> manual might not be the most valuable thing for a user to read. It
> also runs a risk of coming across as a general purpose language which
> may mislead readers.
> 
> I think, a document structure where the important features (app and
> builtin functions, variables and mappers (with examples)) are
> highlighted in the early parts and details such as variable scope,
> expressions, operators etc. are described in the middle and concluded
> with sites specific details might serve well.

You mean like a brief overview of important things / tutorial in the
beginning?

Or do you mean that the language reference should not have a bottom-up
structure, starting with the simple/primary concepts and building up
from that?

Mihael

> 
> --
> Ketan
> 
> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > Hi,
> >
> > Mike pointed out that I should probably post a link to what I'm trying
> > to do with the user guide just in case things are going awfully wrong.
> > So here it is:
> >
> > http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
> >
> > Mike mention K&R a number of times and I think I like that general idea.
> > The above, so far, is a mix between K&R Chapter 2 and Appendix A.
> >
> > The main idea is to keep things concise by separating parallelism
> > behaviour from what a program actually does. In that sense, one would
> > first describe the language, and then go into the details of how a
> > particular implementation goes about running programs.
> >
> > I'm not particularly opposed to starting with a brief tutorial in the
> > spirit of K&R Chapter 1.
> >
> > Mihael
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Mon May 18 15:27:16 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 15:27:16 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <1431976853.12263.13.camel@echo3>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
Message-ID: <CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>

I think a little bit of both.  A brief overview of important things
with examples in chapter 1. Then, topics like "Task Parallel
Computing" or "Many Task Computing" may cover foreach, futures and
task parallelism as implemented in Swift. Then, a chapter like "File
I/O" may cover the mappers, etc. Then a topic like operators and
variables may bring in more of the language reference in that
describing types, structs, etc. Next may be builtin functions.
Followed by a chapter on Compute sites which covers Resource and data
managers, remote access and providers and such.

So, to elaborate here is a rough outline I have in mind:

Chap 1. Swift Overview
-- Importance of Swift
-- Swift example 1
   -- hello world, description, concept show in this example
-- Swift example 2
   -- Run an app functions, describe importance of app functions
-- Swift example 3
   -- Do file mappings, describe mappers
-- Swift example 4
  -- Do foreach loop, describe foreach
-- Swift example 5 (if needed)

Chap 2. Many Task Computing in Swift
-- Parallelism in DAGs
-- Futures
-- foreach loop and variants
-- iterate

Chap 3. File I/O
-- File types
-- Simple file def
-- Complex Mappers
-- Read and Write to Files

Chap 4. Variables and Operators
-- types
-- structs
-- import

Chap 5. User defined and Builtin functions
-- string
-- numerical
-- stat
-- math

Chap 6. Swift providers
-- Coasters
-- Data providers
-- Compute Providers

Chap 6. Remote compute sites
-- Modes of running

Appendix A. Swift Grammar in BNF
Appendix B. A table of Swift conf options
Appendix C. Swift cheatsheet


On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>> I think Swift is a special purpose language where ~20% of features are
>> used for ~80% of cases. Consequently, providing a K&R style language
>> manual might not be the most valuable thing for a user to read. It
>> also runs a risk of coming across as a general purpose language which
>> may mislead readers.
>>
>> I think, a document structure where the important features (app and
>> builtin functions, variables and mappers (with examples)) are
>> highlighted in the early parts and details such as variable scope,
>> expressions, operators etc. are described in the middle and concluded
>> with sites specific details might serve well.
>
> You mean like a brief overview of important things / tutorial in the
> beginning?
>
> Or do you mean that the language reference should not have a bottom-up
> structure, starting with the simple/primary concepts and building up
> from that?
>
> Mihael
>
>>
>> --
>> Ketan
>>
>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> > Hi,
>> >
>> > Mike pointed out that I should probably post a link to what I'm trying
>> > to do with the user guide just in case things are going awfully wrong.
>> > So here it is:
>> >
>> > http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>> >
>> > Mike mention K&R a number of times and I think I like that general idea.
>> > The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>> >
>> > The main idea is to keep things concise by separating parallelism
>> > behaviour from what a program actually does. In that sense, one would
>> > first describe the language, and then go into the details of how a
>> > particular implementation goes about running programs.
>> >
>> > I'm not particularly opposed to starting with a brief tutorial in the
>> > spirit of K&R Chapter 1.
>> >
>> > Mihael
>> >
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From ketan at mcs.anl.gov  Mon May 18 15:33:22 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 15:33:22 -0500
Subject: [Swift-devel] exec not in path despite providing absolute path
In-Reply-To: <1431810694.23991.6.camel@echo3>
References: <CAMUuvioWtK2au3VB26QzeZkdagfZWeT3xgqWURTVFNhDep1tXw@mail.gmail.com>
	<1431810694.23991.6.camel@echo3>
Message-ID: <CAMUuvioK5WKrexZGHB4z38SUx3n3jxyb9Fb30a=hR_xnaoVzLw@mail.gmail.com>

Thanks, this has worked. --Ketan

On Sat, May 16, 2015 at 4:11 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> The magic of hidden includes.
>
> You are not actually using the site you defined. You probably want to
> say
>
> sites: [localhost]
>
> at some point in that config file.
>
> What's happening is that "site: [local]" from etc/swift.conf is used,
> and that does not contain your app.
>
> Mihael
>
> On Fri, 2015-05-15 at 11:56 -0500, Ketan Maheshwari wrote:
>> Hi,
>>
>> I am providing absolute path to an executable but Swift is complaining
>> that it is not in path. Here is the error message:
>>
>> $ swift -config local.conf nek.swift
>> Swift trunk git-rev: 86de22e2841518dbf337a0e0da56ad1dd9dd70f8 heads/master 6343
>> RunID: run004
>> Progress: Fri, 15 May 2015 11:51:55-0500
>> Execution failed:
>> Exception in genrun:
>>     Arguments: [-d, LST.json, -u, LST_f90.tusr, --makenek,
>> ~/maxhutch/nek/makenek, RTI_LST]
>>     Host: local
>>     Directory: nek-run004/jobs/g/genrun-aj3y9t8m
>> exception @ swift-int.k, line: 467
>> swiftStaging:execute2 @ swift.k, line: 139
>> executeSelect @ swift.k, line: 207
>> Caused by: Cannot find executable genrun on site system path
>> throw @ swift-int.k, line: 78
>> checkJobStatus @ swift-int.k, line: 406
>>
>> k:assign @ swift.k, line: 174
>> swift:execute @ nek, line: 46
>> app_genrun @ nek, line: 46
>>
>>
>> Here is the config:
>>
>> $ cat local.conf
>> site.localhost {
>>     execution {
>>         type: "local"
>>         URL: "localhost"
>>     }
>>     filesystem {
>>         type: "local"
>>         URL: "localhost"
>>     }
>>     workDirectory: "/tmp/swift.workdir"
>>     app.cat {
>>         executable: "/bin/cat"
>>     }
>>     app.genrun {
>>         executable: "/home/ketan/maxhutch/nek-tools/genrun/genrun.py"
>>     }
>> }
>>
>> lazyErrors: false
>> executionRetries: 0
>> keepSiteDir: true
>> providerStagingPinSwiftFiles: false
>> alwaysTransferWrapperLog: true
>>
>> The log dir is attached.
>>
>> Any suggestions?
>>
>> Thanks,
>> --
>> Ketan
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Mon May 18 16:18:43 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 May 2015 14:18:43 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
Message-ID: <1431983923.13328.12.camel@echo3>

Right. This is pretty close to what K&R does.

I was personally thinking the following: intro, language reference,
runtime reference, tools reference. It's not trying to sell anything or
emphasize anything. You use it when you need answers. You don't really
read it. You scan it for what you need, and then read the parts that are
relevant to what you are trying to do. The order is from basic concepts
to more complex ones. It's an order that most people would expect,
especially people that have little experience with swift. It is like
that because the purpose is for a user to quickly find the information
they need, not for us to point out to users what we think is cool about
our stuff.

What you describe is fine. And there is no reason why the two things
cannot coexist. Which is the general idea that I think we discussed
during the last meeting. There's formal documentation and there's
informal introductions / tutorials. The latter are different. You read
them sequentially, like a book. The order is the order in which things
are important in swift. You wouldn't start with types and variables.

Mihael

On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
> I think a little bit of both.  A brief overview of important things
> with examples in chapter 1. Then, topics like "Task Parallel
> Computing" or "Many Task Computing" may cover foreach, futures and
> task parallelism as implemented in Swift. Then, a chapter like "File
> I/O" may cover the mappers, etc. Then a topic like operators and
> variables may bring in more of the language reference in that
> describing types, structs, etc. Next may be builtin functions.
> Followed by a chapter on Compute sites which covers Resource and data
> managers, remote access and providers and such.
> 
> So, to elaborate here is a rough outline I have in mind:
> 
> Chap 1. Swift Overview
> -- Importance of Swift
> -- Swift example 1
>    -- hello world, description, concept show in this example
> -- Swift example 2
>    -- Run an app functions, describe importance of app functions
> -- Swift example 3
>    -- Do file mappings, describe mappers
> -- Swift example 4
>   -- Do foreach loop, describe foreach
> -- Swift example 5 (if needed)
> 
> Chap 2. Many Task Computing in Swift
> -- Parallelism in DAGs
> -- Futures
> -- foreach loop and variants
> -- iterate
> 
> Chap 3. File I/O
> -- File types
> -- Simple file def
> -- Complex Mappers
> -- Read and Write to Files
> 
> Chap 4. Variables and Operators
> -- types
> -- structs
> -- import
> 
> Chap 5. User defined and Builtin functions
> -- string
> -- numerical
> -- stat
> -- math
> 
> Chap 6. Swift providers
> -- Coasters
> -- Data providers
> -- Compute Providers
> 
> Chap 6. Remote compute sites
> -- Modes of running
> 
> Appendix A. Swift Grammar in BNF
> Appendix B. A table of Swift conf options
> Appendix C. Swift cheatsheet
> 
> 
> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
> >> I think Swift is a special purpose language where ~20% of features are
> >> used for ~80% of cases. Consequently, providing a K&R style language
> >> manual might not be the most valuable thing for a user to read. It
> >> also runs a risk of coming across as a general purpose language which
> >> may mislead readers.
> >>
> >> I think, a document structure where the important features (app and
> >> builtin functions, variables and mappers (with examples)) are
> >> highlighted in the early parts and details such as variable scope,
> >> expressions, operators etc. are described in the middle and concluded
> >> with sites specific details might serve well.
> >
> > You mean like a brief overview of important things / tutorial in the
> > beginning?
> >
> > Or do you mean that the language reference should not have a bottom-up
> > structure, starting with the simple/primary concepts and building up
> > from that?
> >
> > Mihael
> >
> >>
> >> --
> >> Ketan
> >>
> >> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >> > Hi,
> >> >
> >> > Mike pointed out that I should probably post a link to what I'm trying
> >> > to do with the user guide just in case things are going awfully wrong.
> >> > So here it is:
> >> >
> >> > http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
> >> >
> >> > Mike mention K&R a number of times and I think I like that general idea.
> >> > The above, so far, is a mix between K&R Chapter 2 and Appendix A.
> >> >
> >> > The main idea is to keep things concise by separating parallelism
> >> > behaviour from what a program actually does. In that sense, one would
> >> > first describe the language, and then go into the details of how a
> >> > particular implementation goes about running programs.
> >> >
> >> > I'm not particularly opposed to starting with a brief tutorial in the
> >> > spirit of K&R Chapter 1.
> >> >
> >> > Mihael
> >> >
> >> >
> >> > _______________________________________________
> >> > Swift-devel mailing list
> >> > Swift-devel at ci.uchicago.edu
> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Mon May 18 16:34:15 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 16:34:15 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <1431983923.13328.12.camel@echo3>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
Message-ID: <CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>

This is true except for the first chapter. First chapter is going to
be read by a significantly larger number of people and the rest of it
would be ctrl-f'd. I think we may want to push some stuff further down
in the user guide than what is in the link you originally posted.

--
Ketan

On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> Right. This is pretty close to what K&R does.
>
> I was personally thinking the following: intro, language reference,
> runtime reference, tools reference. It's not trying to sell anything or
> emphasize anything. You use it when you need answers. You don't really
> read it. You scan it for what you need, and then read the parts that are
> relevant to what you are trying to do. The order is from basic concepts
> to more complex ones. It's an order that most people would expect,
> especially people that have little experience with swift. It is like
> that because the purpose is for a user to quickly find the information
> they need, not for us to point out to users what we think is cool about
> our stuff.
>
> What you describe is fine. And there is no reason why the two things
> cannot coexist. Which is the general idea that I think we discussed
> during the last meeting. There's formal documentation and there's
> informal introductions / tutorials. The latter are different. You read
> them sequentially, like a book. The order is the order in which things
> are important in swift. You wouldn't start with types and variables.
>
> Mihael
>
> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
>> I think a little bit of both.  A brief overview of important things
>> with examples in chapter 1. Then, topics like "Task Parallel
>> Computing" or "Many Task Computing" may cover foreach, futures and
>> task parallelism as implemented in Swift. Then, a chapter like "File
>> I/O" may cover the mappers, etc. Then a topic like operators and
>> variables may bring in more of the language reference in that
>> describing types, structs, etc. Next may be builtin functions.
>> Followed by a chapter on Compute sites which covers Resource and data
>> managers, remote access and providers and such.
>>
>> So, to elaborate here is a rough outline I have in mind:
>>
>> Chap 1. Swift Overview
>> -- Importance of Swift
>> -- Swift example 1
>>    -- hello world, description, concept show in this example
>> -- Swift example 2
>>    -- Run an app functions, describe importance of app functions
>> -- Swift example 3
>>    -- Do file mappings, describe mappers
>> -- Swift example 4
>>   -- Do foreach loop, describe foreach
>> -- Swift example 5 (if needed)
>>
>> Chap 2. Many Task Computing in Swift
>> -- Parallelism in DAGs
>> -- Futures
>> -- foreach loop and variants
>> -- iterate
>>
>> Chap 3. File I/O
>> -- File types
>> -- Simple file def
>> -- Complex Mappers
>> -- Read and Write to Files
>>
>> Chap 4. Variables and Operators
>> -- types
>> -- structs
>> -- import
>>
>> Chap 5. User defined and Builtin functions
>> -- string
>> -- numerical
>> -- stat
>> -- math
>>
>> Chap 6. Swift providers
>> -- Coasters
>> -- Data providers
>> -- Compute Providers
>>
>> Chap 6. Remote compute sites
>> -- Modes of running
>>
>> Appendix A. Swift Grammar in BNF
>> Appendix B. A table of Swift conf options
>> Appendix C. Swift cheatsheet
>>
>>
>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> > On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>> >> I think Swift is a special purpose language where ~20% of features are
>> >> used for ~80% of cases. Consequently, providing a K&R style language
>> >> manual might not be the most valuable thing for a user to read. It
>> >> also runs a risk of coming across as a general purpose language which
>> >> may mislead readers.
>> >>
>> >> I think, a document structure where the important features (app and
>> >> builtin functions, variables and mappers (with examples)) are
>> >> highlighted in the early parts and details such as variable scope,
>> >> expressions, operators etc. are described in the middle and concluded
>> >> with sites specific details might serve well.
>> >
>> > You mean like a brief overview of important things / tutorial in the
>> > beginning?
>> >
>> > Or do you mean that the language reference should not have a bottom-up
>> > structure, starting with the simple/primary concepts and building up
>> > from that?
>> >
>> > Mihael
>> >
>> >>
>> >> --
>> >> Ketan
>> >>
>> >> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> >> > Hi,
>> >> >
>> >> > Mike pointed out that I should probably post a link to what I'm trying
>> >> > to do with the user guide just in case things are going awfully wrong.
>> >> > So here it is:
>> >> >
>> >> > http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>> >> >
>> >> > Mike mention K&R a number of times and I think I like that general idea.
>> >> > The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>> >> >
>> >> > The main idea is to keep things concise by separating parallelism
>> >> > behaviour from what a program actually does. In that sense, one would
>> >> > first describe the language, and then go into the details of how a
>> >> > particular implementation goes about running programs.
>> >> >
>> >> > I'm not particularly opposed to starting with a brief tutorial in the
>> >> > spirit of K&R Chapter 1.
>> >> >
>> >> > Mihael
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Swift-devel mailing list
>> >> > Swift-devel at ci.uchicago.edu
>> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Mon May 18 16:42:08 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 May 2015 14:42:08 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
Message-ID: <1431985328.13894.1.camel@echo3>

The link I posted is not the first chapter. It is one piece of the
documentation.

Mihael

On Mon, 2015-05-18 at 16:34 -0500, Ketan Maheshwari wrote:
> This is true except for the first chapter. First chapter is going to
> be read by a significantly larger number of people and the rest of it
> would be ctrl-f'd. I think we may want to push some stuff further down
> in the user guide than what is in the link you originally posted.
> 
> --
> Ketan
> 
> On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > Right. This is pretty close to what K&R does.
> >
> > I was personally thinking the following: intro, language reference,
> > runtime reference, tools reference. It's not trying to sell anything or
> > emphasize anything. You use it when you need answers. You don't really
> > read it. You scan it for what you need, and then read the parts that are
> > relevant to what you are trying to do. The order is from basic concepts
> > to more complex ones. It's an order that most people would expect,
> > especially people that have little experience with swift. It is like
> > that because the purpose is for a user to quickly find the information
> > they need, not for us to point out to users what we think is cool about
> > our stuff.
> >
> > What you describe is fine. And there is no reason why the two things
> > cannot coexist. Which is the general idea that I think we discussed
> > during the last meeting. There's formal documentation and there's
> > informal introductions / tutorials. The latter are different. You read
> > them sequentially, like a book. The order is the order in which things
> > are important in swift. You wouldn't start with types and variables.
> >
> > Mihael
> >
> > On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
> >> I think a little bit of both.  A brief overview of important things
> >> with examples in chapter 1. Then, topics like "Task Parallel
> >> Computing" or "Many Task Computing" may cover foreach, futures and
> >> task parallelism as implemented in Swift. Then, a chapter like "File
> >> I/O" may cover the mappers, etc. Then a topic like operators and
> >> variables may bring in more of the language reference in that
> >> describing types, structs, etc. Next may be builtin functions.
> >> Followed by a chapter on Compute sites which covers Resource and data
> >> managers, remote access and providers and such.
> >>
> >> So, to elaborate here is a rough outline I have in mind:
> >>
> >> Chap 1. Swift Overview
> >> -- Importance of Swift
> >> -- Swift example 1
> >>    -- hello world, description, concept show in this example
> >> -- Swift example 2
> >>    -- Run an app functions, describe importance of app functions
> >> -- Swift example 3
> >>    -- Do file mappings, describe mappers
> >> -- Swift example 4
> >>   -- Do foreach loop, describe foreach
> >> -- Swift example 5 (if needed)
> >>
> >> Chap 2. Many Task Computing in Swift
> >> -- Parallelism in DAGs
> >> -- Futures
> >> -- foreach loop and variants
> >> -- iterate
> >>
> >> Chap 3. File I/O
> >> -- File types
> >> -- Simple file def
> >> -- Complex Mappers
> >> -- Read and Write to Files
> >>
> >> Chap 4. Variables and Operators
> >> -- types
> >> -- structs
> >> -- import
> >>
> >> Chap 5. User defined and Builtin functions
> >> -- string
> >> -- numerical
> >> -- stat
> >> -- math
> >>
> >> Chap 6. Swift providers
> >> -- Coasters
> >> -- Data providers
> >> -- Compute Providers
> >>
> >> Chap 6. Remote compute sites
> >> -- Modes of running
> >>
> >> Appendix A. Swift Grammar in BNF
> >> Appendix B. A table of Swift conf options
> >> Appendix C. Swift cheatsheet
> >>
> >>
> >> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >> > On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
> >> >> I think Swift is a special purpose language where ~20% of features are
> >> >> used for ~80% of cases. Consequently, providing a K&R style language
> >> >> manual might not be the most valuable thing for a user to read. It
> >> >> also runs a risk of coming across as a general purpose language which
> >> >> may mislead readers.
> >> >>
> >> >> I think, a document structure where the important features (app and
> >> >> builtin functions, variables and mappers (with examples)) are
> >> >> highlighted in the early parts and details such as variable scope,
> >> >> expressions, operators etc. are described in the middle and concluded
> >> >> with sites specific details might serve well.
> >> >
> >> > You mean like a brief overview of important things / tutorial in the
> >> > beginning?
> >> >
> >> > Or do you mean that the language reference should not have a bottom-up
> >> > structure, starting with the simple/primary concepts and building up
> >> > from that?
> >> >
> >> > Mihael
> >> >
> >> >>
> >> >> --
> >> >> Ketan
> >> >>
> >> >> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > Mike pointed out that I should probably post a link to what I'm trying
> >> >> > to do with the user guide just in case things are going awfully wrong.
> >> >> > So here it is:
> >> >> >
> >> >> > http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
> >> >> >
> >> >> > Mike mention K&R a number of times and I think I like that general idea.
> >> >> > The above, so far, is a mix between K&R Chapter 2 and Appendix A.
> >> >> >
> >> >> > The main idea is to keep things concise by separating parallelism
> >> >> > behaviour from what a program actually does. In that sense, one would
> >> >> > first describe the language, and then go into the details of how a
> >> >> > particular implementation goes about running programs.
> >> >> >
> >> >> > I'm not particularly opposed to starting with a brief tutorial in the
> >> >> > spirit of K&R Chapter 1.
> >> >> >
> >> >> > Mihael
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Swift-devel mailing list
> >> >> > Swift-devel at ci.uchicago.edu
> >> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >> >
> >> >
> >> > _______________________________________________
> >> > Swift-devel mailing list
> >> > Swift-devel at ci.uchicago.edu
> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From yadunand at uchicago.edu  Mon May 18 16:48:01 2015
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Mon, 18 May 2015 16:48:01 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>	<1431976853.12263.13.camel@echo3>	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
Message-ID: <555A5E11.9090603@uchicago.edu>

The overview sounds a lot like the tutorial. I think it's better to have 
tutorial-like material just go into the tutorial rather than just get 
duplicated in the userguide.
If the tutorial does not give the user an adequate picture of items in 
Ketan's overview, the tutorial needs fixing.

The webpage points users to the tutorial, and if the tutorial isn't 
adequately covering the useful 20% of the language, I suspect that they 
wouldn't bother to get to the userguide.

Thanks,
Yadu

On 05/18/2015 04:34 PM, Ketan Maheshwari wrote:
> This is true except for the first chapter. First chapter is going to
> be read by a significantly larger number of people and the rest of it
> would be ctrl-f'd. I think we may want to push some stuff further down
> in the user guide than what is in the link you originally posted.
>
> --
> Ketan
>
> On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> Right. This is pretty close to what K&R does.
>>
>> I was personally thinking the following: intro, language reference,
>> runtime reference, tools reference. It's not trying to sell anything or
>> emphasize anything. You use it when you need answers. You don't really
>> read it. You scan it for what you need, and then read the parts that are
>> relevant to what you are trying to do. The order is from basic concepts
>> to more complex ones. It's an order that most people would expect,
>> especially people that have little experience with swift. It is like
>> that because the purpose is for a user to quickly find the information
>> they need, not for us to point out to users what we think is cool about
>> our stuff.
>>
>> What you describe is fine. And there is no reason why the two things
>> cannot coexist. Which is the general idea that I think we discussed
>> during the last meeting. There's formal documentation and there's
>> informal introductions / tutorials. The latter are different. You read
>> them sequentially, like a book. The order is the order in which things
>> are important in swift. You wouldn't start with types and variables.
>>
>> Mihael
>>
>> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
>>> I think a little bit of both.  A brief overview of important things
>>> with examples in chapter 1. Then, topics like "Task Parallel
>>> Computing" or "Many Task Computing" may cover foreach, futures and
>>> task parallelism as implemented in Swift. Then, a chapter like "File
>>> I/O" may cover the mappers, etc. Then a topic like operators and
>>> variables may bring in more of the language reference in that
>>> describing types, structs, etc. Next may be builtin functions.
>>> Followed by a chapter on Compute sites which covers Resource and data
>>> managers, remote access and providers and such.
>>>
>>> So, to elaborate here is a rough outline I have in mind:
>>>
>>> Chap 1. Swift Overview
>>> -- Importance of Swift
>>> -- Swift example 1
>>>     -- hello world, description, concept show in this example
>>> -- Swift example 2
>>>     -- Run an app functions, describe importance of app functions
>>> -- Swift example 3
>>>     -- Do file mappings, describe mappers
>>> -- Swift example 4
>>>    -- Do foreach loop, describe foreach
>>> -- Swift example 5 (if needed)
>>>
>>> Chap 2. Many Task Computing in Swift
>>> -- Parallelism in DAGs
>>> -- Futures
>>> -- foreach loop and variants
>>> -- iterate
>>>
>>> Chap 3. File I/O
>>> -- File types
>>> -- Simple file def
>>> -- Complex Mappers
>>> -- Read and Write to Files
>>>
>>> Chap 4. Variables and Operators
>>> -- types
>>> -- structs
>>> -- import
>>>
>>> Chap 5. User defined and Builtin functions
>>> -- string
>>> -- numerical
>>> -- stat
>>> -- math
>>>
>>> Chap 6. Swift providers
>>> -- Coasters
>>> -- Data providers
>>> -- Compute Providers
>>>
>>> Chap 6. Remote compute sites
>>> -- Modes of running
>>>
>>> Appendix A. Swift Grammar in BNF
>>> Appendix B. A table of Swift conf options
>>> Appendix C. Swift cheatsheet
>>>
>>>
>>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>>>>> I think Swift is a special purpose language where ~20% of features are
>>>>> used for ~80% of cases. Consequently, providing a K&R style language
>>>>> manual might not be the most valuable thing for a user to read. It
>>>>> also runs a risk of coming across as a general purpose language which
>>>>> may mislead readers.
>>>>>
>>>>> I think, a document structure where the important features (app and
>>>>> builtin functions, variables and mappers (with examples)) are
>>>>> highlighted in the early parts and details such as variable scope,
>>>>> expressions, operators etc. are described in the middle and concluded
>>>>> with sites specific details might serve well.
>>>> You mean like a brief overview of important things / tutorial in the
>>>> beginning?
>>>>
>>>> Or do you mean that the language reference should not have a bottom-up
>>>> structure, starting with the simple/primary concepts and building up
>>>> from that?
>>>>
>>>> Mihael
>>>>
>>>>> --
>>>>> Ketan
>>>>>
>>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Mike pointed out that I should probably post a link to what I'm trying
>>>>>> to do with the user guide just in case things are going awfully wrong.
>>>>>> So here it is:
>>>>>>
>>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>>>>>>
>>>>>> Mike mention K&R a number of times and I think I like that general idea.
>>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>>>>>>
>>>>>> The main idea is to keep things concise by separating parallelism
>>>>>> behaviour from what a program actually does. In that sense, one would
>>>>>> first describe the language, and then go into the details of how a
>>>>>> particular implementation goes about running programs.
>>>>>>
>>>>>> I'm not particularly opposed to starting with a brief tutorial in the
>>>>>> spirit of K&R Chapter 1.
>>>>>>
>>>>>> Mihael
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From hategan at mcs.anl.gov  Mon May 18 16:52:19 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 May 2015 14:52:19 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <555A5E11.9090603@uchicago.edu>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
	<555A5E11.9090603@uchicago.edu>
Message-ID: <1431985939.14029.1.camel@echo3>

On Mon, 2015-05-18 at 16:48 -0500, Yadu Nand Babuji wrote:
> The overview sounds a lot like the tutorial. I think it's better to have 
> tutorial-like material just go into the tutorial rather than just get 
> duplicated in the userguide.
> If the tutorial does not give the user an adequate picture of items in 
> Ketan's overview, the tutorial needs fixing.

The tutorial can link to the more formal documentation. It's a form of
"hey, if you really want to know...". 

Mihael

> 
> The webpage points users to the tutorial, and if the tutorial isn't 
> adequately covering the useful 20% of the language, I suspect that they 
> wouldn't bother to get to the userguide.
> 
> Thanks,
> Yadu
> 
> On 05/18/2015 04:34 PM, Ketan Maheshwari wrote:
> > This is true except for the first chapter. First chapter is going to
> > be read by a significantly larger number of people and the rest of it
> > would be ctrl-f'd. I think we may want to push some stuff further down
> > in the user guide than what is in the link you originally posted.
> >
> > --
> > Ketan
> >
> > On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >> Right. This is pretty close to what K&R does.
> >>
> >> I was personally thinking the following: intro, language reference,
> >> runtime reference, tools reference. It's not trying to sell anything or
> >> emphasize anything. You use it when you need answers. You don't really
> >> read it. You scan it for what you need, and then read the parts that are
> >> relevant to what you are trying to do. The order is from basic concepts
> >> to more complex ones. It's an order that most people would expect,
> >> especially people that have little experience with swift. It is like
> >> that because the purpose is for a user to quickly find the information
> >> they need, not for us to point out to users what we think is cool about
> >> our stuff.
> >>
> >> What you describe is fine. And there is no reason why the two things
> >> cannot coexist. Which is the general idea that I think we discussed
> >> during the last meeting. There's formal documentation and there's
> >> informal introductions / tutorials. The latter are different. You read
> >> them sequentially, like a book. The order is the order in which things
> >> are important in swift. You wouldn't start with types and variables.
> >>
> >> Mihael
> >>
> >> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
> >>> I think a little bit of both.  A brief overview of important things
> >>> with examples in chapter 1. Then, topics like "Task Parallel
> >>> Computing" or "Many Task Computing" may cover foreach, futures and
> >>> task parallelism as implemented in Swift. Then, a chapter like "File
> >>> I/O" may cover the mappers, etc. Then a topic like operators and
> >>> variables may bring in more of the language reference in that
> >>> describing types, structs, etc. Next may be builtin functions.
> >>> Followed by a chapter on Compute sites which covers Resource and data
> >>> managers, remote access and providers and such.
> >>>
> >>> So, to elaborate here is a rough outline I have in mind:
> >>>
> >>> Chap 1. Swift Overview
> >>> -- Importance of Swift
> >>> -- Swift example 1
> >>>     -- hello world, description, concept show in this example
> >>> -- Swift example 2
> >>>     -- Run an app functions, describe importance of app functions
> >>> -- Swift example 3
> >>>     -- Do file mappings, describe mappers
> >>> -- Swift example 4
> >>>    -- Do foreach loop, describe foreach
> >>> -- Swift example 5 (if needed)
> >>>
> >>> Chap 2. Many Task Computing in Swift
> >>> -- Parallelism in DAGs
> >>> -- Futures
> >>> -- foreach loop and variants
> >>> -- iterate
> >>>
> >>> Chap 3. File I/O
> >>> -- File types
> >>> -- Simple file def
> >>> -- Complex Mappers
> >>> -- Read and Write to Files
> >>>
> >>> Chap 4. Variables and Operators
> >>> -- types
> >>> -- structs
> >>> -- import
> >>>
> >>> Chap 5. User defined and Builtin functions
> >>> -- string
> >>> -- numerical
> >>> -- stat
> >>> -- math
> >>>
> >>> Chap 6. Swift providers
> >>> -- Coasters
> >>> -- Data providers
> >>> -- Compute Providers
> >>>
> >>> Chap 6. Remote compute sites
> >>> -- Modes of running
> >>>
> >>> Appendix A. Swift Grammar in BNF
> >>> Appendix B. A table of Swift conf options
> >>> Appendix C. Swift cheatsheet
> >>>
> >>>
> >>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
> >>>>> I think Swift is a special purpose language where ~20% of features are
> >>>>> used for ~80% of cases. Consequently, providing a K&R style language
> >>>>> manual might not be the most valuable thing for a user to read. It
> >>>>> also runs a risk of coming across as a general purpose language which
> >>>>> may mislead readers.
> >>>>>
> >>>>> I think, a document structure where the important features (app and
> >>>>> builtin functions, variables and mappers (with examples)) are
> >>>>> highlighted in the early parts and details such as variable scope,
> >>>>> expressions, operators etc. are described in the middle and concluded
> >>>>> with sites specific details might serve well.
> >>>> You mean like a brief overview of important things / tutorial in the
> >>>> beginning?
> >>>>
> >>>> Or do you mean that the language reference should not have a bottom-up
> >>>> structure, starting with the simple/primary concepts and building up
> >>>> from that?
> >>>>
> >>>> Mihael
> >>>>
> >>>>> --
> >>>>> Ketan
> >>>>>
> >>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Mike pointed out that I should probably post a link to what I'm trying
> >>>>>> to do with the user guide just in case things are going awfully wrong.
> >>>>>> So here it is:
> >>>>>>
> >>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
> >>>>>>
> >>>>>> Mike mention K&R a number of times and I think I like that general idea.
> >>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
> >>>>>>
> >>>>>> The main idea is to keep things concise by separating parallelism
> >>>>>> behaviour from what a program actually does. In that sense, one would
> >>>>>> first describe the language, and then go into the details of how a
> >>>>>> particular implementation goes about running programs.
> >>>>>>
> >>>>>> I'm not particularly opposed to starting with a brief tutorial in the
> >>>>>> spirit of K&R Chapter 1.
> >>>>>>
> >>>>>> Mihael
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Swift-devel mailing list
> >>>>>> Swift-devel at ci.uchicago.edu
> >>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>>
> >>>> _______________________________________________
> >>>> Swift-devel mailing list
> >>>> Swift-devel at ci.uchicago.edu
> >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Mon May 18 17:04:32 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 17:04:32 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <555A5E11.9090603@uchicago.edu>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
	<555A5E11.9090603@uchicago.edu>
Message-ID: <CAMUuvirOGu_Bd7y3OSqSSwkZduA2ehHpgZutgx65+nFLhcbyhQ@mail.gmail.com>

I would argue in favor of some redundancy between user-guide and
tutorial. Assuming that most users are busy (and impatient), they will
randomly click one link and we will have a better chance to get the
point across.

I think we should collaboratively develop a ToC for user-guide and
tutorial before filling in the details. I think most details exist
already, it is a matter of figuring the placements.

--
Ketan

On Mon, May 18, 2015 at 4:48 PM, Yadu Nand Babuji <yadunand at uchicago.edu> wrote:
> The overview sounds a lot like the tutorial. I think it's better to have
> tutorial-like material just go into the tutorial rather than just get
> duplicated in the userguide.
> If the tutorial does not give the user an adequate picture of items in
> Ketan's overview, the tutorial needs fixing.
>
> The webpage points users to the tutorial, and if the tutorial isn't
> adequately covering the useful 20% of the language, I suspect that they
> wouldn't bother to get to the userguide.
>
> Thanks,
> Yadu
>
> On 05/18/2015 04:34 PM, Ketan Maheshwari wrote:
>> This is true except for the first chapter. First chapter is going to
>> be read by a significantly larger number of people and the rest of it
>> would be ctrl-f'd. I think we may want to push some stuff further down
>> in the user guide than what is in the link you originally posted.
>>
>> --
>> Ketan
>>
>> On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>> Right. This is pretty close to what K&R does.
>>>
>>> I was personally thinking the following: intro, language reference,
>>> runtime reference, tools reference. It's not trying to sell anything or
>>> emphasize anything. You use it when you need answers. You don't really
>>> read it. You scan it for what you need, and then read the parts that are
>>> relevant to what you are trying to do. The order is from basic concepts
>>> to more complex ones. It's an order that most people would expect,
>>> especially people that have little experience with swift. It is like
>>> that because the purpose is for a user to quickly find the information
>>> they need, not for us to point out to users what we think is cool about
>>> our stuff.
>>>
>>> What you describe is fine. And there is no reason why the two things
>>> cannot coexist. Which is the general idea that I think we discussed
>>> during the last meeting. There's formal documentation and there's
>>> informal introductions / tutorials. The latter are different. You read
>>> them sequentially, like a book. The order is the order in which things
>>> are important in swift. You wouldn't start with types and variables.
>>>
>>> Mihael
>>>
>>> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
>>>> I think a little bit of both.  A brief overview of important things
>>>> with examples in chapter 1. Then, topics like "Task Parallel
>>>> Computing" or "Many Task Computing" may cover foreach, futures and
>>>> task parallelism as implemented in Swift. Then, a chapter like "File
>>>> I/O" may cover the mappers, etc. Then a topic like operators and
>>>> variables may bring in more of the language reference in that
>>>> describing types, structs, etc. Next may be builtin functions.
>>>> Followed by a chapter on Compute sites which covers Resource and data
>>>> managers, remote access and providers and such.
>>>>
>>>> So, to elaborate here is a rough outline I have in mind:
>>>>
>>>> Chap 1. Swift Overview
>>>> -- Importance of Swift
>>>> -- Swift example 1
>>>>     -- hello world, description, concept show in this example
>>>> -- Swift example 2
>>>>     -- Run an app functions, describe importance of app functions
>>>> -- Swift example 3
>>>>     -- Do file mappings, describe mappers
>>>> -- Swift example 4
>>>>    -- Do foreach loop, describe foreach
>>>> -- Swift example 5 (if needed)
>>>>
>>>> Chap 2. Many Task Computing in Swift
>>>> -- Parallelism in DAGs
>>>> -- Futures
>>>> -- foreach loop and variants
>>>> -- iterate
>>>>
>>>> Chap 3. File I/O
>>>> -- File types
>>>> -- Simple file def
>>>> -- Complex Mappers
>>>> -- Read and Write to Files
>>>>
>>>> Chap 4. Variables and Operators
>>>> -- types
>>>> -- structs
>>>> -- import
>>>>
>>>> Chap 5. User defined and Builtin functions
>>>> -- string
>>>> -- numerical
>>>> -- stat
>>>> -- math
>>>>
>>>> Chap 6. Swift providers
>>>> -- Coasters
>>>> -- Data providers
>>>> -- Compute Providers
>>>>
>>>> Chap 6. Remote compute sites
>>>> -- Modes of running
>>>>
>>>> Appendix A. Swift Grammar in BNF
>>>> Appendix B. A table of Swift conf options
>>>> Appendix C. Swift cheatsheet
>>>>
>>>>
>>>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>>>>>> I think Swift is a special purpose language where ~20% of features are
>>>>>> used for ~80% of cases. Consequently, providing a K&R style language
>>>>>> manual might not be the most valuable thing for a user to read. It
>>>>>> also runs a risk of coming across as a general purpose language which
>>>>>> may mislead readers.
>>>>>>
>>>>>> I think, a document structure where the important features (app and
>>>>>> builtin functions, variables and mappers (with examples)) are
>>>>>> highlighted in the early parts and details such as variable scope,
>>>>>> expressions, operators etc. are described in the middle and concluded
>>>>>> with sites specific details might serve well.
>>>>> You mean like a brief overview of important things / tutorial in the
>>>>> beginning?
>>>>>
>>>>> Or do you mean that the language reference should not have a bottom-up
>>>>> structure, starting with the simple/primary concepts and building up
>>>>> from that?
>>>>>
>>>>> Mihael
>>>>>
>>>>>> --
>>>>>> Ketan
>>>>>>
>>>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Mike pointed out that I should probably post a link to what I'm trying
>>>>>>> to do with the user guide just in case things are going awfully wrong.
>>>>>>> So here it is:
>>>>>>>
>>>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>>>>>>>
>>>>>>> Mike mention K&R a number of times and I think I like that general idea.
>>>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>>>>>>>
>>>>>>> The main idea is to keep things concise by separating parallelism
>>>>>>> behaviour from what a program actually does. In that sense, one would
>>>>>>> first describe the language, and then go into the details of how a
>>>>>>> particular implementation goes about running programs.
>>>>>>>
>>>>>>> I'm not particularly opposed to starting with a brief tutorial in the
>>>>>>> spirit of K&R Chapter 1.
>>>>>>>
>>>>>>> Mihael
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Swift-devel mailing list
>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>>>
>>>>> _______________________________________________
>>>>> Swift-devel mailing list
>>>>> Swift-devel at ci.uchicago.edu
>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel


From hategan at mcs.anl.gov  Mon May 18 17:50:40 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Mon, 18 May 2015 15:50:40 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuvirOGu_Bd7y3OSqSSwkZduA2ehHpgZutgx65+nFLhcbyhQ@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
	<555A5E11.9090603@uchicago.edu>
	<CAMUuvirOGu_Bd7y3OSqSSwkZduA2ehHpgZutgx65+nFLhcbyhQ@mail.gmail.com>
Message-ID: <1431989440.14403.14.camel@echo3>

On Mon, 2015-05-18 at 17:04 -0500, Ketan Maheshwari wrote:
> I would argue in favor of some redundancy between user-guide and
> tutorial. Assuming that most users are busy (and impatient), they will
> randomly click one link and we will have a better chance to get the
> point across.

I'm not sure what you are saying. Can you be more specific? With
examples perhaps?

> 
> I think we should collaboratively develop a ToC for user-guide and
> tutorial before filling in the details. I think most details exist
> already, it is a matter of figuring the placements.

As long as we are on the same page about what we want to achieve, then I
think that's a good idea.

But I disagree that the details are there and that this is a matter of
shuffling paragraphs around. The details we have are lousy, the grammar
is lousy, and many things are incorrect. I can give many examples, but
let's just pick one:

-------------
Sequential iteration can be expressed using the iterate construct:

step[0] = initialCondition();
iterate ix {
  step[ix] = simulate(step[ix-1]);
}
This fragment will initialise the 0-th element of the step array to some
initial condition, and then repeatedly run the simulate procedure, using
each execution?s outputs as input to the next step.
-----------

The code doesn't work (it's missing the "until"), the step array is not
a condition array (it can be anything). The code also doesn't work
because it will deadlock on step[-1], because the initial ix is 0.

This snippet is in "Arrays and parallel execution", except iterate does
not do parallel execution. The first paragraph of that section reads:

-------------
Arrays of values can be declared using the [] suffix. Following is an
example of an array of strings:

string pets[] = ["shane", "noddy", "leo"];
An array may be mapped to a collection of files, one element per file,
by using a different form of mapping expression. For example, the
filesys_mapper maps all files matching a particular unix glob pattern
into an array:
-------------

What's a "different form of mapping expression"? (the context for that
is many paragraphs earlier). There is absolutely no mention of parallel
execution or parallelism in the section titled "Arrays and Parallel
Execution".

Furthermore, Swift has evolved a lot since that documentation was
written, and very little of that is documented. So no, the details are
not there.

Mihael

> 
> --
> Ketan
> 
> On Mon, May 18, 2015 at 4:48 PM, Yadu Nand Babuji <yadunand at uchicago.edu> wrote:
> > The overview sounds a lot like the tutorial. I think it's better to have
> > tutorial-like material just go into the tutorial rather than just get
> > duplicated in the userguide.
> > If the tutorial does not give the user an adequate picture of items in
> > Ketan's overview, the tutorial needs fixing.
> >
> > The webpage points users to the tutorial, and if the tutorial isn't
> > adequately covering the useful 20% of the language, I suspect that they
> > wouldn't bother to get to the userguide.
> >
> > Thanks,
> > Yadu
> >
> > On 05/18/2015 04:34 PM, Ketan Maheshwari wrote:
> >> This is true except for the first chapter. First chapter is going to
> >> be read by a significantly larger number of people and the rest of it
> >> would be ctrl-f'd. I think we may want to push some stuff further down
> >> in the user guide than what is in the link you originally posted.
> >>
> >> --
> >> Ketan
> >>
> >> On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>> Right. This is pretty close to what K&R does.
> >>>
> >>> I was personally thinking the following: intro, language reference,
> >>> runtime reference, tools reference. It's not trying to sell anything or
> >>> emphasize anything. You use it when you need answers. You don't really
> >>> read it. You scan it for what you need, and then read the parts that are
> >>> relevant to what you are trying to do. The order is from basic concepts
> >>> to more complex ones. It's an order that most people would expect,
> >>> especially people that have little experience with swift. It is like
> >>> that because the purpose is for a user to quickly find the information
> >>> they need, not for us to point out to users what we think is cool about
> >>> our stuff.
> >>>
> >>> What you describe is fine. And there is no reason why the two things
> >>> cannot coexist. Which is the general idea that I think we discussed
> >>> during the last meeting. There's formal documentation and there's
> >>> informal introductions / tutorials. The latter are different. You read
> >>> them sequentially, like a book. The order is the order in which things
> >>> are important in swift. You wouldn't start with types and variables.
> >>>
> >>> Mihael
> >>>
> >>> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
> >>>> I think a little bit of both.  A brief overview of important things
> >>>> with examples in chapter 1. Then, topics like "Task Parallel
> >>>> Computing" or "Many Task Computing" may cover foreach, futures and
> >>>> task parallelism as implemented in Swift. Then, a chapter like "File
> >>>> I/O" may cover the mappers, etc. Then a topic like operators and
> >>>> variables may bring in more of the language reference in that
> >>>> describing types, structs, etc. Next may be builtin functions.
> >>>> Followed by a chapter on Compute sites which covers Resource and data
> >>>> managers, remote access and providers and such.
> >>>>
> >>>> So, to elaborate here is a rough outline I have in mind:
> >>>>
> >>>> Chap 1. Swift Overview
> >>>> -- Importance of Swift
> >>>> -- Swift example 1
> >>>>     -- hello world, description, concept show in this example
> >>>> -- Swift example 2
> >>>>     -- Run an app functions, describe importance of app functions
> >>>> -- Swift example 3
> >>>>     -- Do file mappings, describe mappers
> >>>> -- Swift example 4
> >>>>    -- Do foreach loop, describe foreach
> >>>> -- Swift example 5 (if needed)
> >>>>
> >>>> Chap 2. Many Task Computing in Swift
> >>>> -- Parallelism in DAGs
> >>>> -- Futures
> >>>> -- foreach loop and variants
> >>>> -- iterate
> >>>>
> >>>> Chap 3. File I/O
> >>>> -- File types
> >>>> -- Simple file def
> >>>> -- Complex Mappers
> >>>> -- Read and Write to Files
> >>>>
> >>>> Chap 4. Variables and Operators
> >>>> -- types
> >>>> -- structs
> >>>> -- import
> >>>>
> >>>> Chap 5. User defined and Builtin functions
> >>>> -- string
> >>>> -- numerical
> >>>> -- stat
> >>>> -- math
> >>>>
> >>>> Chap 6. Swift providers
> >>>> -- Coasters
> >>>> -- Data providers
> >>>> -- Compute Providers
> >>>>
> >>>> Chap 6. Remote compute sites
> >>>> -- Modes of running
> >>>>
> >>>> Appendix A. Swift Grammar in BNF
> >>>> Appendix B. A table of Swift conf options
> >>>> Appendix C. Swift cheatsheet
> >>>>
> >>>>
> >>>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
> >>>>>> I think Swift is a special purpose language where ~20% of features are
> >>>>>> used for ~80% of cases. Consequently, providing a K&R style language
> >>>>>> manual might not be the most valuable thing for a user to read. It
> >>>>>> also runs a risk of coming across as a general purpose language which
> >>>>>> may mislead readers.
> >>>>>>
> >>>>>> I think, a document structure where the important features (app and
> >>>>>> builtin functions, variables and mappers (with examples)) are
> >>>>>> highlighted in the early parts and details such as variable scope,
> >>>>>> expressions, operators etc. are described in the middle and concluded
> >>>>>> with sites specific details might serve well.
> >>>>> You mean like a brief overview of important things / tutorial in the
> >>>>> beginning?
> >>>>>
> >>>>> Or do you mean that the language reference should not have a bottom-up
> >>>>> structure, starting with the simple/primary concepts and building up
> >>>>> from that?
> >>>>>
> >>>>> Mihael
> >>>>>
> >>>>>> --
> >>>>>> Ketan
> >>>>>>
> >>>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> Mike pointed out that I should probably post a link to what I'm trying
> >>>>>>> to do with the user guide just in case things are going awfully wrong.
> >>>>>>> So here it is:
> >>>>>>>
> >>>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
> >>>>>>>
> >>>>>>> Mike mention K&R a number of times and I think I like that general idea.
> >>>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
> >>>>>>>
> >>>>>>> The main idea is to keep things concise by separating parallelism
> >>>>>>> behaviour from what a program actually does. In that sense, one would
> >>>>>>> first describe the language, and then go into the details of how a
> >>>>>>> particular implementation goes about running programs.
> >>>>>>>
> >>>>>>> I'm not particularly opposed to starting with a brief tutorial in the
> >>>>>>> spirit of K&R Chapter 1.
> >>>>>>>
> >>>>>>> Mihael
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> Swift-devel mailing list
> >>>>>>> Swift-devel at ci.uchicago.edu
> >>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>>>
> >>>>> _______________________________________________
> >>>>> Swift-devel mailing list
> >>>>> Swift-devel at ci.uchicago.edu
> >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>
> >>> _______________________________________________
> >>> Swift-devel mailing list
> >>> Swift-devel at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From ketan at mcs.anl.gov  Mon May 18 18:18:31 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Mon, 18 May 2015 18:18:31 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <cbb21e392c0c4b78be56c2f9c6ee8e2f@LUCKMAN.anl.gov>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<1431983923.13328.12.camel@echo3>
	<CAMUuvipnOYRbx0YEVYs6Y_nYJPNFFkzAB6aPnH_12suRygw5CA@mail.gmail.com>
	<555A5E11.9090603@uchicago.edu>
	<CAMUuvirOGu_Bd7y3OSqSSwkZduA2ehHpgZutgx65+nFLhcbyhQ@mail.gmail.com>
	<cbb21e392c0c4b78be56c2f9c6ee8e2f@LUCKMAN.anl.gov>
Message-ID: <CAMUuvip17wH2reb0qciJRBc397MagNGUqWbOEE37MZTd3FFa7g@mail.gmail.com>

On Mon, May 18, 2015 at 5:50 PM, Hategan-Marandiuc, Philip M.
<hategan at mcs.anl.gov> wrote:
> On Mon, 2015-05-18 at 17:04 -0500, Ketan Maheshwari wrote:
>> I would argue in favor of some redundancy between user-guide and
>> tutorial. Assuming that most users are busy (and impatient), they will
>> randomly click one link and we will have a better chance to get the
>> point across.
>
> I'm not sure what you are saying. Can you be more specific? With
> examples perhaps?

I mean the tutorial and user-guide need not be strictly mutually
exclusive. At least, the initial sections in both could have some
overlapping, eg. example scripts. Later sections may gradually branch
out into "what+why" for the user-guide and  "how to" for the tutorial.

>
>>
>> I think we should collaboratively develop a ToC for user-guide and
>> tutorial before filling in the details. I think most details exist
>> already, it is a matter of figuring the placements.
>
> As long as we are on the same page about what we want to achieve, then I
> think that's a good idea.
>
> But I disagree that the details are there and that this is a matter of
> shuffling paragraphs around. The details we have are lousy, the grammar
> is lousy, and many things are incorrect. I can give many examples, but
> let's just pick one:
>
> -------------
> Sequential iteration can be expressed using the iterate construct:
>
> step[0] = initialCondition();
> iterate ix {
>   step[ix] = simulate(step[ix-1]);
> }
> This fragment will initialise the 0-th element of the step array to some
> initial condition, and then repeatedly run the simulate procedure, using
> each execution?s outputs as input to the next step.
> -----------
>
> The code doesn't work (it's missing the "until"), the step array is not
> a condition array (it can be anything). The code also doesn't work
> because it will deadlock on step[-1], because the initial ix is 0.
>
> This snippet is in "Arrays and parallel execution", except iterate does
> not do parallel execution. The first paragraph of that section reads:
>
> -------------
> Arrays of values can be declared using the [] suffix. Following is an
> example of an array of strings:
>
> string pets[] = ["shane", "noddy", "leo"];
> An array may be mapped to a collection of files, one element per file,
> by using a different form of mapping expression. For example, the
> filesys_mapper maps all files matching a particular unix glob pattern
> into an array:
> -------------
>
> What's a "different form of mapping expression"? (the context for that
> is many paragraphs earlier). There is absolutely no mention of parallel
> execution or parallelism in the section titled "Arrays and Parallel
> Execution".
>
> Furthermore, Swift has evolved a lot since that documentation was
> written, and very little of that is documented. So no, the details are
> not there.

Ok, agree.

>
> Mihael
>
>>
>> --
>> Ketan
>>
>> On Mon, May 18, 2015 at 4:48 PM, Yadu Nand Babuji <yadunand at uchicago.edu> wrote:
>> > The overview sounds a lot like the tutorial. I think it's better to have
>> > tutorial-like material just go into the tutorial rather than just get
>> > duplicated in the userguide.
>> > If the tutorial does not give the user an adequate picture of items in
>> > Ketan's overview, the tutorial needs fixing.
>> >
>> > The webpage points users to the tutorial, and if the tutorial isn't
>> > adequately covering the useful 20% of the language, I suspect that they
>> > wouldn't bother to get to the userguide.
>> >
>> > Thanks,
>> > Yadu
>> >
>> > On 05/18/2015 04:34 PM, Ketan Maheshwari wrote:
>> >> This is true except for the first chapter. First chapter is going to
>> >> be read by a significantly larger number of people and the rest of it
>> >> would be ctrl-f'd. I think we may want to push some stuff further down
>> >> in the user guide than what is in the link you originally posted.
>> >>
>> >> --
>> >> Ketan
>> >>
>> >> On Mon, May 18, 2015 at 4:18 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> >>> Right. This is pretty close to what K&R does.
>> >>>
>> >>> I was personally thinking the following: intro, language reference,
>> >>> runtime reference, tools reference. It's not trying to sell anything or
>> >>> emphasize anything. You use it when you need answers. You don't really
>> >>> read it. You scan it for what you need, and then read the parts that are
>> >>> relevant to what you are trying to do. The order is from basic concepts
>> >>> to more complex ones. It's an order that most people would expect,
>> >>> especially people that have little experience with swift. It is like
>> >>> that because the purpose is for a user to quickly find the information
>> >>> they need, not for us to point out to users what we think is cool about
>> >>> our stuff.
>> >>>
>> >>> What you describe is fine. And there is no reason why the two things
>> >>> cannot coexist. Which is the general idea that I think we discussed
>> >>> during the last meeting. There's formal documentation and there's
>> >>> informal introductions / tutorials. The latter are different. You read
>> >>> them sequentially, like a book. The order is the order in which things
>> >>> are important in swift. You wouldn't start with types and variables.
>> >>>
>> >>> Mihael
>> >>>
>> >>> On Mon, 2015-05-18 at 15:27 -0500, Ketan Maheshwari wrote:
>> >>>> I think a little bit of both.  A brief overview of important things
>> >>>> with examples in chapter 1. Then, topics like "Task Parallel
>> >>>> Computing" or "Many Task Computing" may cover foreach, futures and
>> >>>> task parallelism as implemented in Swift. Then, a chapter like "File
>> >>>> I/O" may cover the mappers, etc. Then a topic like operators and
>> >>>> variables may bring in more of the language reference in that
>> >>>> describing types, structs, etc. Next may be builtin functions.
>> >>>> Followed by a chapter on Compute sites which covers Resource and data
>> >>>> managers, remote access and providers and such.
>> >>>>
>> >>>> So, to elaborate here is a rough outline I have in mind:
>> >>>>
>> >>>> Chap 1. Swift Overview
>> >>>> -- Importance of Swift
>> >>>> -- Swift example 1
>> >>>>     -- hello world, description, concept show in this example
>> >>>> -- Swift example 2
>> >>>>     -- Run an app functions, describe importance of app functions
>> >>>> -- Swift example 3
>> >>>>     -- Do file mappings, describe mappers
>> >>>> -- Swift example 4
>> >>>>    -- Do foreach loop, describe foreach
>> >>>> -- Swift example 5 (if needed)
>> >>>>
>> >>>> Chap 2. Many Task Computing in Swift
>> >>>> -- Parallelism in DAGs
>> >>>> -- Futures
>> >>>> -- foreach loop and variants
>> >>>> -- iterate
>> >>>>
>> >>>> Chap 3. File I/O
>> >>>> -- File types
>> >>>> -- Simple file def
>> >>>> -- Complex Mappers
>> >>>> -- Read and Write to Files
>> >>>>
>> >>>> Chap 4. Variables and Operators
>> >>>> -- types
>> >>>> -- structs
>> >>>> -- import
>> >>>>
>> >>>> Chap 5. User defined and Builtin functions
>> >>>> -- string
>> >>>> -- numerical
>> >>>> -- stat
>> >>>> -- math
>> >>>>
>> >>>> Chap 6. Swift providers
>> >>>> -- Coasters
>> >>>> -- Data providers
>> >>>> -- Compute Providers
>> >>>>
>> >>>> Chap 6. Remote compute sites
>> >>>> -- Modes of running
>> >>>>
>> >>>> Appendix A. Swift Grammar in BNF
>> >>>> Appendix B. A table of Swift conf options
>> >>>> Appendix C. Swift cheatsheet
>> >>>>
>> >>>>
>> >>>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> >>>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>> >>>>>> I think Swift is a special purpose language where ~20% of features are
>> >>>>>> used for ~80% of cases. Consequently, providing a K&R style language
>> >>>>>> manual might not be the most valuable thing for a user to read. It
>> >>>>>> also runs a risk of coming across as a general purpose language which
>> >>>>>> may mislead readers.
>> >>>>>>
>> >>>>>> I think, a document structure where the important features (app and
>> >>>>>> builtin functions, variables and mappers (with examples)) are
>> >>>>>> highlighted in the early parts and details such as variable scope,
>> >>>>>> expressions, operators etc. are described in the middle and concluded
>> >>>>>> with sites specific details might serve well.
>> >>>>> You mean like a brief overview of important things / tutorial in the
>> >>>>> beginning?
>> >>>>>
>> >>>>> Or do you mean that the language reference should not have a bottom-up
>> >>>>> structure, starting with the simple/primary concepts and building up
>> >>>>> from that?
>> >>>>>
>> >>>>> Mihael
>> >>>>>
>> >>>>>> --
>> >>>>>> Ketan
>> >>>>>>
>> >>>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>> Mike pointed out that I should probably post a link to what I'm trying
>> >>>>>>> to do with the user guide just in case things are going awfully wrong.
>> >>>>>>> So here it is:
>> >>>>>>>
>> >>>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>> >>>>>>>
>> >>>>>>> Mike mention K&R a number of times and I think I like that general idea.
>> >>>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>> >>>>>>>
>> >>>>>>> The main idea is to keep things concise by separating parallelism
>> >>>>>>> behaviour from what a program actually does. In that sense, one would
>> >>>>>>> first describe the language, and then go into the details of how a
>> >>>>>>> particular implementation goes about running programs.
>> >>>>>>>
>> >>>>>>> I'm not particularly opposed to starting with a brief tutorial in the
>> >>>>>>> spirit of K&R Chapter 1.
>> >>>>>>>
>> >>>>>>> Mihael
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> Swift-devel mailing list
>> >>>>>>> Swift-devel at ci.uchicago.edu
>> >>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Swift-devel mailing list
>> >>>>> Swift-devel at ci.uchicago.edu
>> >>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >>>
>> >>> _______________________________________________
>> >>> Swift-devel mailing list
>> >>> Swift-devel at ci.uchicago.edu
>> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >> _______________________________________________
>> >> Swift-devel mailing list
>> >> Swift-devel at ci.uchicago.edu
>> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


From ketan at mcs.anl.gov  Tue May 19 07:27:54 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Tue, 19 May 2015 07:27:54 -0500
Subject: [Swift-devel] Updating swift-lang for 0.96.0
In-Reply-To: <CAMUuvip3X7iVUcDfOSyL6sjSmNWsyjw9P9NVuQ5K5dV3w7yF2Q@mail.gmail.com>
References: <554CDEFF.8010808@uchicago.edu>
	<CAMUuvip3X7iVUcDfOSyL6sjSmNWsyjw9P9NVuQ5K5dV3w7yF2Q@mail.gmail.com>
Message-ID: <CAMUuviqe8CGLpG-77uuQ35d4G9PaRvDTK5Cj9G9rWKw1Z8p1yg@mail.gmail.com>

I think having Swift 0.96 and 0.95 related things (Download,
Documents) on the same page is confusing.

Many sites have a separate page for older versions of tools and the
link points to pages dedicated to those versions.

With Swift, we can perhaps have everything related to 0.95 stashed
into a top-level directory swift-0.95 and stash everything in there.
Then we create a page olderversions.html and link it there. Similarly,
link /main to the latest/stable version and keep things related to
that version in the pages. I think it would look cleaner.

--
Ketan

On Fri, May 8, 2015 at 12:40 PM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:
> Looks good to me.
>
> Comments:
>
> -- in the release number we had two parts x.xx (0.95) whereas in this one it
> is x.xx.x (0.96.0). Intentional?
> -- Release notes points to 404 not found (may be disable link for now?)
> -- Configuration file format link points to your home www area, may be point
> to github conf schema link
> (https://github.com/swift-lang/swift-k/blob/master/resources/swift.conf.schema)
> or swift-lang.org area.
>
>
>
> On Fri, May 8, 2015 at 11:06 AM, Yadu Nand Babuji <yadunand at uchicago.edu>
> wrote:
>>
>> Hi All,
>>
>> I'm updating swift-lang.org for the 0.96.0 release, and should be done
>> in an hour.
>> I would really appreciate it if folks could try links and go over the
>> website text to
>> check if everything is in order.
>>
>> Thanks,
>> Yadu
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>


From lpesce at uchicago.edu  Tue May 19 09:25:44 2015
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Tue, 19 May 2015 09:25:44 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
Message-ID: <F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>

I will throw in my useless 2 cents. I have been part of five or six swift based projects
all involving what I would call large computations (at least a million core hours, usually tens of terabytes of data).

In my experience:
1) The basic variables are all we ever needed.
2) The basic control loops is all we ever needed even if some more complex ones could have been handy (they just weren?t there when I started).
3) All the file system trickeries known to the devil and his friends are absolutely necessary. I never found a project that didn?t end up chocking on its data.
4) All the tricks for dealing intelligently with staging of data and code and ?offloading? control are essential to avoid the hatred of the system administrators and other users.
5) A very clear understanding of where Swift is robust and where it isn?t are very important because nobody likes to have a 300 node run come crashing down on you creating an infestation of zombies that eventually might have forced Beagle?s admin to reboot. This is not criticism, I participated heavily in writing that script, so I am on the accused team. 

I can try to be more helpful, if you want me to. 

I have worked on a number of parallel projects, where essentially the same workflow was implemented with and without swift and both teams played to win. That is one paper I plan to write some day?


> On May 18, 2015, at 3:27 PM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:
> 
> I think a little bit of both.  A brief overview of important things
> with examples in chapter 1. Then, topics like "Task Parallel
> Computing" or "Many Task Computing" may cover foreach, futures and
> task parallelism as implemented in Swift. Then, a chapter like "File
> I/O" may cover the mappers, etc. Then a topic like operators and
> variables may bring in more of the language reference in that
> describing types, structs, etc. Next may be builtin functions.
> Followed by a chapter on Compute sites which covers Resource and data
> managers, remote access and providers and such.
> 
> So, to elaborate here is a rough outline I have in mind:
> 
> Chap 1. Swift Overview
> -- Importance of Swift
> -- Swift example 1
>   -- hello world, description, concept show in this example
> -- Swift example 2
>   -- Run an app functions, describe importance of app functions
> -- Swift example 3
>   -- Do file mappings, describe mappers
> -- Swift example 4
>  -- Do foreach loop, describe foreach
> -- Swift example 5 (if needed)
> 
> Chap 2. Many Task Computing in Swift
> -- Parallelism in DAGs
> -- Futures
> -- foreach loop and variants
> -- iterate
> 
> Chap 3. File I/O
> -- File types
> -- Simple file def
> -- Complex Mappers
> -- Read and Write to Files
> 
> Chap 4. Variables and Operators
> -- types
> -- structs
> -- import
> 
> Chap 5. User defined and Builtin functions
> -- string
> -- numerical
> -- stat
> -- math
> 
> Chap 6. Swift providers
> -- Coasters
> -- Data providers
> -- Compute Providers
> 
> Chap 6. Remote compute sites
> -- Modes of running
> 
> Appendix A. Swift Grammar in BNF
> Appendix B. A table of Swift conf options
> Appendix C. Swift cheatsheet
> 
> 
> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>>> I think Swift is a special purpose language where ~20% of features are
>>> used for ~80% of cases. Consequently, providing a K&R style language
>>> manual might not be the most valuable thing for a user to read. It
>>> also runs a risk of coming across as a general purpose language which
>>> may mislead readers.
>>> 
>>> I think, a document structure where the important features (app and
>>> builtin functions, variables and mappers (with examples)) are
>>> highlighted in the early parts and details such as variable scope,
>>> expressions, operators etc. are described in the middle and concluded
>>> with sites specific details might serve well.
>> 
>> You mean like a brief overview of important things / tutorial in the
>> beginning?
>> 
>> Or do you mean that the language reference should not have a bottom-up
>> structure, starting with the simple/primary concepts and building up
>> from that?
>> 
>> Mihael
>> 
>>> 
>>> --
>>> Ketan
>>> 
>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>> Hi,
>>>> 
>>>> Mike pointed out that I should probably post a link to what I'm trying
>>>> to do with the user guide just in case things are going awfully wrong.
>>>> So here it is:
>>>> 
>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>>>> 
>>>> Mike mention K&R a number of times and I think I like that general idea.
>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>>>> 
>>>> The main idea is to keep things concise by separating parallelism
>>>> behaviour from what a program actually does. In that sense, one would
>>>> first describe the language, and then go into the details of how a
>>>> particular implementation goes about running programs.
>>>> 
>>>> I'm not particularly opposed to starting with a brief tutorial in the
>>>> spirit of K&R Chapter 1.
>>>> 
>>>> Mihael
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From ketan at mcs.anl.gov  Tue May 19 10:14:20 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Tue, 19 May 2015 10:14:20 -0500
Subject: [Swift-devel] doc pull-edit-test-push workflow
Message-ID: <CAMUuvip2D3O4sd=uBip-nxATE7F3Lj7dWbZAdzG=xnV8Y49XvQ@mail.gmail.com>

Hi,

Would there be interest in automating the workflow for editing Swift
documentation. I am thinking of a script that eases the following:

1. pull the latest docs from git
2. Edit
3. Build
4. Test in local browser
5. Commit and push to git
6. Push to www

I think we discussed a need for this kind of script in the past.

--
Ketan


From wilde at anl.gov  Tue May 19 10:49:05 2015
From: wilde at anl.gov (Michael Wilde)
Date: Tue, 19 May 2015 10:49:05 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
References: <1431810281.23991.3.camel@echo3>	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>	<1431976853.12263.13.camel@echo3>	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
Message-ID: <555B5B71.7040705@anl.gov>

Thanks, Lorenzo - this is very useful feedback.

We have some good documentation for 0.96.0 in progress, and will welcome 
further feedback, testing and improvements on it once its posted for 
comment.

- Mike

On 5/19/15 9:25 AM, Lorenzo Pesce wrote:
> I will throw in my useless 2 cents. I have been part of five or six swift based projects
> all involving what I would call large computations (at least a million core hours, usually tens of terabytes of data).
>
> In my experience:
> 1) The basic variables are all we ever needed.
> 2) The basic control loops is all we ever needed even if some more complex ones could have been handy (they just weren?t there when I started).
> 3) All the file system trickeries known to the devil and his friends are absolutely necessary. I never found a project that didn?t end up chocking on its data.
> 4) All the tricks for dealing intelligently with staging of data and code and ?offloading? control are essential to avoid the hatred of the system administrators and other users.
> 5) A very clear understanding of where Swift is robust and where it isn?t are very important because nobody likes to have a 300 node run come crashing down on you creating an infestation of zombies that eventually might have forced Beagle?s admin to reboot. This is not criticism, I participated heavily in writing that script, so I am on the accused team.
>
> I can try to be more helpful, if you want me to.
>
> I have worked on a number of parallel projects, where essentially the same workflow was implemented with and without swift and both teams played to win. That is one paper I plan to write some day?
>
>
>> On May 18, 2015, at 3:27 PM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:
>>
>> I think a little bit of both.  A brief overview of important things
>> with examples in chapter 1. Then, topics like "Task Parallel
>> Computing" or "Many Task Computing" may cover foreach, futures and
>> task parallelism as implemented in Swift. Then, a chapter like "File
>> I/O" may cover the mappers, etc. Then a topic like operators and
>> variables may bring in more of the language reference in that
>> describing types, structs, etc. Next may be builtin functions.
>> Followed by a chapter on Compute sites which covers Resource and data
>> managers, remote access and providers and such.
>>
>> So, to elaborate here is a rough outline I have in mind:
>>
>> Chap 1. Swift Overview
>> -- Importance of Swift
>> -- Swift example 1
>>    -- hello world, description, concept show in this example
>> -- Swift example 2
>>    -- Run an app functions, describe importance of app functions
>> -- Swift example 3
>>    -- Do file mappings, describe mappers
>> -- Swift example 4
>>   -- Do foreach loop, describe foreach
>> -- Swift example 5 (if needed)
>>
>> Chap 2. Many Task Computing in Swift
>> -- Parallelism in DAGs
>> -- Futures
>> -- foreach loop and variants
>> -- iterate
>>
>> Chap 3. File I/O
>> -- File types
>> -- Simple file def
>> -- Complex Mappers
>> -- Read and Write to Files
>>
>> Chap 4. Variables and Operators
>> -- types
>> -- structs
>> -- import
>>
>> Chap 5. User defined and Builtin functions
>> -- string
>> -- numerical
>> -- stat
>> -- math
>>
>> Chap 6. Swift providers
>> -- Coasters
>> -- Data providers
>> -- Compute Providers
>>
>> Chap 6. Remote compute sites
>> -- Modes of running
>>
>> Appendix A. Swift Grammar in BNF
>> Appendix B. A table of Swift conf options
>> Appendix C. Swift cheatsheet
>>
>>
>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>>>> I think Swift is a special purpose language where ~20% of features are
>>>> used for ~80% of cases. Consequently, providing a K&R style language
>>>> manual might not be the most valuable thing for a user to read. It
>>>> also runs a risk of coming across as a general purpose language which
>>>> may mislead readers.
>>>>
>>>> I think, a document structure where the important features (app and
>>>> builtin functions, variables and mappers (with examples)) are
>>>> highlighted in the early parts and details such as variable scope,
>>>> expressions, operators etc. are described in the middle and concluded
>>>> with sites specific details might serve well.
>>> You mean like a brief overview of important things / tutorial in the
>>> beginning?
>>>
>>> Or do you mean that the language reference should not have a bottom-up
>>> structure, starting with the simple/primary concepts and building up
>>> from that?
>>>
>>> Mihael
>>>
>>>> --
>>>> Ketan
>>>>
>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>> Hi,
>>>>>
>>>>> Mike pointed out that I should probably post a link to what I'm trying
>>>>> to do with the user guide just in case things are going awfully wrong.
>>>>> So here it is:
>>>>>
>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>>>>>
>>>>> Mike mention K&R a number of times and I think I like that general idea.
>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>>>>>
>>>>> The main idea is to keep things concise by separating parallelism
>>>>> behaviour from what a program actually does. In that sense, one would
>>>>> first describe the language, and then go into the details of how a
>>>>> particular implementation goes about running programs.
>>>>>
>>>>> I'm not particularly opposed to starting with a brief tutorial in the
>>>>> spirit of K&R Chapter 1.
>>>>>
>>>>> Mihael
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Swift-devel mailing list
>>>>> Swift-devel at ci.uchicago.edu
>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago



From lpesce at uchicago.edu  Tue May 19 10:53:12 2015
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Tue, 19 May 2015 10:53:12 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <555B5B71.7040705@anl.gov>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
	<555B5B71.7040705@anl.gov>
Message-ID: <800F6F03-26FF-4CC0-9654-774081362E97@uchicago.edu>

Thanks Mike.

I am also going to start planning the workflow lectures on Beagle, goes without saying that swift will be part of it ;-)

I will start sending draft when I have something. If you have any ideas or pointers, please send them over.

Lorenzo

> On May 19, 2015, at 10:49 AM, Michael Wilde <wilde at anl.gov> wrote:
> 
> Thanks, Lorenzo - this is very useful feedback.
> 
> We have some good documentation for 0.96.0 in progress, and will welcome 
> further feedback, testing and improvements on it once its posted for 
> comment.
> 
> - Mike
> 
> On 5/19/15 9:25 AM, Lorenzo Pesce wrote:
>> I will throw in my useless 2 cents. I have been part of five or six swift based projects
>> all involving what I would call large computations (at least a million core hours, usually tens of terabytes of data).
>> 
>> In my experience:
>> 1) The basic variables are all we ever needed.
>> 2) The basic control loops is all we ever needed even if some more complex ones could have been handy (they just weren?t there when I started).
>> 3) All the file system trickeries known to the devil and his friends are absolutely necessary. I never found a project that didn?t end up chocking on its data.
>> 4) All the tricks for dealing intelligently with staging of data and code and ?offloading? control are essential to avoid the hatred of the system administrators and other users.
>> 5) A very clear understanding of where Swift is robust and where it isn?t are very important because nobody likes to have a 300 node run come crashing down on you creating an infestation of zombies that eventually might have forced Beagle?s admin to reboot. This is not criticism, I participated heavily in writing that script, so I am on the accused team.
>> 
>> I can try to be more helpful, if you want me to.
>> 
>> I have worked on a number of parallel projects, where essentially the same workflow was implemented with and without swift and both teams played to win. That is one paper I plan to write some day?
>> 
>> 
>>> On May 18, 2015, at 3:27 PM, Ketan Maheshwari <ketan at mcs.anl.gov> wrote:
>>> 
>>> I think a little bit of both.  A brief overview of important things
>>> with examples in chapter 1. Then, topics like "Task Parallel
>>> Computing" or "Many Task Computing" may cover foreach, futures and
>>> task parallelism as implemented in Swift. Then, a chapter like "File
>>> I/O" may cover the mappers, etc. Then a topic like operators and
>>> variables may bring in more of the language reference in that
>>> describing types, structs, etc. Next may be builtin functions.
>>> Followed by a chapter on Compute sites which covers Resource and data
>>> managers, remote access and providers and such.
>>> 
>>> So, to elaborate here is a rough outline I have in mind:
>>> 
>>> Chap 1. Swift Overview
>>> -- Importance of Swift
>>> -- Swift example 1
>>>   -- hello world, description, concept show in this example
>>> -- Swift example 2
>>>   -- Run an app functions, describe importance of app functions
>>> -- Swift example 3
>>>   -- Do file mappings, describe mappers
>>> -- Swift example 4
>>>  -- Do foreach loop, describe foreach
>>> -- Swift example 5 (if needed)
>>> 
>>> Chap 2. Many Task Computing in Swift
>>> -- Parallelism in DAGs
>>> -- Futures
>>> -- foreach loop and variants
>>> -- iterate
>>> 
>>> Chap 3. File I/O
>>> -- File types
>>> -- Simple file def
>>> -- Complex Mappers
>>> -- Read and Write to Files
>>> 
>>> Chap 4. Variables and Operators
>>> -- types
>>> -- structs
>>> -- import
>>> 
>>> Chap 5. User defined and Builtin functions
>>> -- string
>>> -- numerical
>>> -- stat
>>> -- math
>>> 
>>> Chap 6. Swift providers
>>> -- Coasters
>>> -- Data providers
>>> -- Compute Providers
>>> 
>>> Chap 6. Remote compute sites
>>> -- Modes of running
>>> 
>>> Appendix A. Swift Grammar in BNF
>>> Appendix B. A table of Swift conf options
>>> Appendix C. Swift cheatsheet
>>> 
>>> 
>>> On Mon, May 18, 2015 at 2:20 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>> On Mon, 2015-05-18 at 13:27 -0500, Ketan Maheshwari wrote:
>>>>> I think Swift is a special purpose language where ~20% of features are
>>>>> used for ~80% of cases. Consequently, providing a K&R style language
>>>>> manual might not be the most valuable thing for a user to read. It
>>>>> also runs a risk of coming across as a general purpose language which
>>>>> may mislead readers.
>>>>> 
>>>>> I think, a document structure where the important features (app and
>>>>> builtin functions, variables and mappers (with examples)) are
>>>>> highlighted in the early parts and details such as variable scope,
>>>>> expressions, operators etc. are described in the middle and concluded
>>>>> with sites specific details might serve well.
>>>> You mean like a brief overview of important things / tutorial in the
>>>> beginning?
>>>> 
>>>> Or do you mean that the language reference should not have a bottom-up
>>>> structure, starting with the simple/primary concepts and building up
>>>> from that?
>>>> 
>>>> Mihael
>>>> 
>>>>> --
>>>>> Ketan
>>>>> 
>>>>> On Sat, May 16, 2015 at 4:04 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> Mike pointed out that I should probably post a link to what I'm trying
>>>>>> to do with the user guide just in case things are going awfully wrong.
>>>>>> So here it is:
>>>>>> 
>>>>>> http://www.mcs.anl.gov/~hategan/ug/ug/ug.html
>>>>>> 
>>>>>> Mike mention K&R a number of times and I think I like that general idea.
>>>>>> The above, so far, is a mix between K&R Chapter 2 and Appendix A.
>>>>>> 
>>>>>> The main idea is to keep things concise by separating parallelism
>>>>>> behaviour from what a program actually does. In that sense, one would
>>>>>> first describe the language, and then go into the details of how a
>>>>>> particular implementation goes about running programs.
>>>>>> 
>>>>>> I'm not particularly opposed to starting with a brief tutorial in the
>>>>>> spirit of K&R Chapter 1.
>>>>>> 
>>>>>> Mihael
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Swift-devel mailing list
>>>>>> Swift-devel at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>> 
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> -- 
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel



From hategan at mcs.anl.gov  Tue May 19 13:34:21 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 19 May 2015 11:34:21 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
Message-ID: <1432060461.25169.2.camel@echo3>

On Tue, 2015-05-19 at 09:25 -0500, Lorenzo Pesce wrote:
> I will throw in my useless 2 cents. I have been part of five or six swift based projects
> all involving what I would call large computations (at least a million core hours, usually tens of terabytes of data).
> 
> In my experience:
> 1) The basic variables are all we ever needed.
> 2) The basic control loops is all we ever needed even if some more complex ones could have been handy (they just weren?t there when I started).
> 3) All the file system trickeries known to the devil and his friends are absolutely necessary. I never found a project that didn?t end up chocking on its data.
> 4) All the tricks for dealing intelligently with staging of data and code and ?offloading? control are essential to avoid the hatred of the system administrators and other users.
> 5) A very clear understanding of where Swift is robust and where it
> isn?t are very important because nobody likes to have a 300 node run
> come crashing down on you creating an infestation of zombies that
> eventually might have forced Beagle?s admin to reboot. This is not
> criticism, I participated heavily in writing that script, so I am on
> the accused team. 

Thanks! That is very useful. Do you have specific feedback on the
documentation? Such as #1 and #2 most annoying things about it?

> 
> I can try to be more helpful, if you want me to. 
> 
> I have worked on a number of parallel projects, where essentially the
> same workflow was implemented with and without swift and both teams
> played to win. That is one paper I plan to write some day?

Wait, who won? :)

Mihael



From lpesce at uchicago.edu  Tue May 19 14:16:33 2015
From: lpesce at uchicago.edu (Lorenzo Pesce)
Date: Tue, 19 May 2015 14:16:33 -0500
Subject: [Swift-devel] user guide work
In-Reply-To: <1432060461.25169.2.camel@echo3>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
	<1432060461.25169.2.camel@echo3>
Message-ID: <9C6AC652-8283-44CC-A91A-8F331998945D@uchicago.edu>


> On May 19, 2015, at 1:34 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> 
> On Tue, 2015-05-19 at 09:25 -0500, Lorenzo Pesce wrote:
>> I will throw in my useless 2 cents. I have been part of five or six swift based projects
>> all involving what I would call large computations (at least a million core hours, usually tens of terabytes of data).
>> 
>> In my experience:
>> 1) The basic variables are all we ever needed.
>> 2) The basic control loops is all we ever needed even if some more complex ones could have been handy (they just weren?t there when I started).
>> 3) All the file system trickeries known to the devil and his friends are absolutely necessary. I never found a project that didn?t end up chocking on its data.
>> 4) All the tricks for dealing intelligently with staging of data and code and ?offloading? control are essential to avoid the hatred of the system administrators and other users.
>> 5) A very clear understanding of where Swift is robust and where it
>> isn?t are very important because nobody likes to have a 300 node run
>> come crashing down on you creating an infestation of zombies that
>> eventually might have forced Beagle?s admin to reboot. This is not
>> criticism, I participated heavily in writing that script, so I am on
>> the accused team. 
> 
> Thanks! That is very useful. Do you have specific feedback on the
> documentation? Such as #1 and #2 most annoying things about it?

Do you want an honest answer? ;-)

My first though were, bear with the humor I mean it in a positive way (all what sounds obnoxious is just my devious sense of humor, promised):

-Awesome, now I know another language in which to write ?Hello World?, gee, I can even have a file called hello world or use tr to make it caps!!!! How do I run 300 genomes which are going to clog up half of my Cray, break my disks and drive my users insane? 

-Hmm? my scheduler choked and returned me errors that are incomprehensible and I am left with a 10 GB log file that requires tea leaves to be sorted out. If I wanted to write something that I can run on 10 nodes I would have used bash. OK, how do I sort out the mess? Is there any procedure to try and figure out what did I do wrong?

-Oh shit, I did not realize that I was making swift stage 10 TB of data during each run. Luckily my admin missed me when he tried to shoot me. :-)

- Can anyone tell me how does swift control the scheduler and keep track of what is running and what should be run? How do I help it sort out what to stage and when to start doing it? Can one help me sort that out without having to read the entire guide? It really looks like my scheduler is comatose and I am being blamed for anything that happens there. ;-)


>> I can try to be more helpful, if you want me to. 
>> 
>> I have worked on a number of parallel projects, where essentially the
>> same workflow was implemented with and without swift and both teams
>> played to win. That is one paper I plan to write some day?
> 
> Wait, who won? :)


Actually the jury is still out and my inclination would be to say that there is a lot to learn from it on how to make using Swift easy on the users who don?t want to learn how to use swift.
I like the swift implementation better (honestly, I am not trying to flatter you), but the other paper was published a year ago.

> 
> Mihael
> 



From hategan at mcs.anl.gov  Tue May 19 18:00:31 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 19 May 2015 16:00:31 -0700
Subject: [Swift-devel] user guide work
In-Reply-To: <9C6AC652-8283-44CC-A91A-8F331998945D@uchicago.edu>
References: <1431810281.23991.3.camel@echo3>
	<CAMUuvio79wZ9=0t5-8KVO6f=hSAr1zG4Mme=MzrM0UHggZ7ONw@mail.gmail.com>
	<1431976853.12263.13.camel@echo3>
	<CAMUuviqsSs37z=icxvPM1cugtN+JLVZYxC7PUdM0+-D9BqFQAQ@mail.gmail.com>
	<F3346DB8-C8D2-45A7-BB4A-80DF59E56891@uchicago.edu>
	<1432060461.25169.2.camel@echo3>
	<9C6AC652-8283-44CC-A91A-8F331998945D@uchicago.edu>
Message-ID: <1432076431.31243.7.camel@echo3>

On Tue, 2015-05-19 at 14:16 -0500, Lorenzo Pesce wrote:
> > On May 19, 2015, at 1:34 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > [...]
> > 
> > Thanks! That is very useful. Do you have specific feedback on the
> > documentation? Such as #1 and #2 most annoying things about it?
> 
> Do you want an honest answer? ;-)

What good is a dishonest answer?

> 
> My first though were, bear with the humor I mean it in a positive way
> (all what sounds obnoxious is just my devious sense of humor,
> promised):
> 
> -Awesome, now I know another language in which to write ?Hello World?,
> gee, I can even have a file called hello world or use tr to make it
> caps!!!! How do I run 300 genomes which are going to clog up half of
> my Cray, break my disks and drive my users insane? 
> 
> -Hmm? my scheduler choked and returned me errors that are
> incomprehensible and I am left with a 10 GB log file that requires tea
> leaves to be sorted out. If I wanted to write something that I can run
> on 10 nodes I would have used bash. OK, how do I sort out the mess? Is
> there any procedure to try and figure out what did I do wrong?
> 
> -Oh shit, I did not realize that I was making swift stage 10 TB of
> data during each run. Luckily my admin missed me when he tried to
> shoot me. :-)
> 
> - Can anyone tell me how does swift control the scheduler and keep
> track of what is running and what should be run? How do I help it sort
> out what to stage and when to start doing it? Can one help me sort
> that out without having to read the entire guide? It really looks like
> my scheduler is comatose and I am being blamed for anything that
> happens there. ;-)

Right. The eternal problem of conceptually similar environments/clusters
all slightly different in devious ways. You can abstract away the
functionality. You cannot abstract away all the ways in which things can
go wrong.

> 
> 
> >> I can try to be more helpful, if you want me to. 
> >> 
> >> I have worked on a number of parallel projects, where essentially the
> >> same workflow was implemented with and without swift and both teams
> >> played to win. That is one paper I plan to write some day?
> > 
> > Wait, who won? :)
> 
> 
> Actually the jury is still out and my inclination would be to say that
> there is a lot to learn from it on how to make using Swift easy on the
> users who don?t want to learn how to use swift.
> I like the swift implementation better (honestly, I am not trying to
> flatter you), but the other paper was published a year ago.

I think we would all appreciate details here.

Mihael 



From ketan at mcs.anl.gov  Thu May 21 10:36:43 2015
From: ketan at mcs.anl.gov (Ketan Maheshwari)
Date: Thu, 21 May 2015 10:36:43 -0500
Subject: [Swift-devel] cd into and run Swift statements in a dir
Message-ID: <CAMUuvioKBL+CJdRoRuC8VmUnJBNj7jbyoxTC0X3Z307_5zYqOQ@mail.gmail.com>

Hi All,

Max Hutchinson and I are working on an application whose first stage
involves building the inputs using an application provided makefile.

(Max, feel free to chime in if I captured things inaccurately from our
discussion)

The makefile must work from a directory where it expects an input
config and will output the executable and a couple of additional
configuration files which will be used in the further stages.

Max (Max, correct me if this is not the case) would normally do this
from within a python script by starting with a "mkdir" and "cd"
command followed by running the makefile and ending with a "cd ..".

After discussing a bit about how to do this in Swift, I was unsure how
we can tell Swift to change the working directory into a user specific
one and run the subsequent statements in there at Swift level.
Interestingly, at runtime, Swift will indeed create its own unique
directory and _Swiftwrap will run things from within this directory.

We discussed a couple of alternatives: a) invoke the makefile by
specifying a target dirname which is created as part of Swift mappers
operations. b) create a tiny wrapper around the makefile and put the
mkdir; cd, cd .. in this wrapper. Max however thinks that while these
are plausible solutions, he would prefer to keep things simpler by not
having to do this if Swift can do it for him.

So, a couple of ideas to achieve this: 1) A way to tell Swift to haul
the swift.work run directories to user area and 2) Have ability to
write script commands in Swift app definition eg. (..) app (..) {
mkdir tmp.$$; cd tmp.$$; appname args}

Thoughts?

Thanks,
Ketan


From yadunand at uchicago.edu  Thu May 28 14:01:41 2015
From: yadunand at uchicago.edu (Yadu Nand Babuji)
Date: Thu, 28 May 2015 14:01:41 -0500
Subject: [Swift-devel] Placing and recovering results from Tcl memory on
	worker ranks
Message-ID: <55676615.7070302@uchicago.edu>

Hi Justin, Tim,

Is there any method available to execute a task per worker rank at the 
end of the run ?
I remember that a cleanup script that could be specified using an 
environment variable that
would be executed as the workers finalized. I'm trying to use node local 
memory by aggregating
results from computation in tcl variables, but I do not know how I could 
specify a leaf app that
would return the results from the tcl-variables per rank at the 
finalizing stage.

Here's what the leaf-code that accumulates in a tcl variable, and this 
appears to work:

@dispatch=WORKER
(float a) accum (float s) "turbine" "0.0" [
"""
if ![info exists var] {set var 0}
set var [expr {$var + <<s>>}]
set <<a>> $var
"""
];

This came from the different approaches I tried to improve the 
performance of the benchmarking code
Ioan's students were using. The bottleneck there was writes to an array, 
which is what I'm trying to avoid
entirely by putting results into on-node memory and then doing a final 
aggregate over all ranks. But
ensuring that the aggregation over ranks happen after all other work has 
completed without specifying
any data dependencies is proving to be a hard problem

Thanks,
Yadu


From wozniak at mcs.anl.gov  Thu May 28 14:20:12 2015
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Thu, 28 May 2015 14:20:12 -0500
Subject: [Swift-devel] Placing and recovering results from Tcl memory on
 worker ranks
In-Reply-To: <55676615.7070302@uchicago.edu>
References: <55676615.7070302@uchicago.edu>
Message-ID: <55676A6C.4000805@mcs.anl.gov>


This kind of cleanup script does not trigger dataflow progress.

On 5/28/2015 2:01 PM, Yadu Nand Babuji wrote:
> Hi Justin, Tim,
>
> Is there any method available to execute a task per worker rank at the
> end of the run ?
> I remember that a cleanup script that could be specified using an
> environment variable that
> would be executed as the workers finalized. I'm trying to use node local
> memory by aggregating
> results from computation in tcl variables, but I do not know how I could
> specify a leaf app that
> would return the results from the tcl-variables per rank at the
> finalizing stage.
>
> Here's what the leaf-code that accumulates in a tcl variable, and this
> appears to work:
>
> @dispatch=WORKER
> (float a) accum (float s) "turbine" "0.0" [
> """
> if ![info exists var] {set var 0}
> set var [expr {$var + <<s>>}]
> set <<a>> $var
> """
> ];
>
> This came from the different approaches I tried to improve the
> performance of the benchmarking code
> Ioan's students were using. The bottleneck there was writes to an array,
> which is what I'm trying to avoid
> entirely by putting results into on-node memory and then doing a final
> aggregate over all ranks. But
> ensuring that the aggregation over ranks happen after all other work has
> completed without specifying
> any data dependencies is proving to be a hard problem
>
> Thanks,
> Yadu
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Justin M Wozniak



From hategan at mcs.anl.gov  Fri May 29 18:46:02 2015
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Fri, 29 May 2015 16:46:02 -0700
Subject: [Swift-devel] the switch statement
Message-ID: <1432943162.6976.6.camel@echo3>

So I have some bad news. The switch statement is broken.

In most c-like languages, the case expressions are restricted to
literals. The reason is that it is the only way to reliably check for
conflicting cases. Allowing arbitrary expressions means that you can't
assert that the cases are non-overlapping at compile-time.

Swift allows expressions in cases. Furthermore, it implements the cases
as 

if (expr == case1expr) {statements1}
else if (expr == case2expr) {statements2}
...

This means that when two cases have the same value, the second one will
be quietly ignored.

We have two options I think:

1. Remove the else. Run statements in all cases that have the right
value
2. Restrict case expressions to literals and check that there are no two
cases with the same value at compile-time.

Thoughts?

Mihael



From tim.g.armstrong at gmail.com  Fri May 29 19:26:25 2015
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Fri, 29 May 2015 19:26:25 -0500
Subject: [Swift-devel] the switch statement
In-Reply-To: <1432943162.6976.6.camel@echo3>
References: <1432943162.6976.6.camel@echo3>
Message-ID: <556903b5.91106b0a.4b3f.ffffe041@mx.google.com>

How about keep it as is but if two cases have the same constant value generate an error/warning? I feel like if you're using non-constant expressions in case labels the programmer is responsible for dealing with overlapping cases.

It seems sensible to me to run the first matching case or default if none match. Running multiple cases seems counter intuitive.

 Swift/t doesn't support expressions in switch anyway if that matters.

-----Original Message-----
From: "Mihael Hategan" <hategan at mcs.anl.gov>
Sent: ?5/?29/?2015 6:46 PM
To: "Swift Devel" <swift-devel at ci.uchicago.edu>
Subject: [Swift-devel] the switch statement

So I have some bad news. The switch statement is broken.

In most c-like languages, the case expressions are restricted to
literals. The reason is that it is the only way to reliably check for
conflicting cases. Allowing arbitrary expressions means that you can't
assert that the cases are non-overlapping at compile-time.

Swift allows expressions in cases. Furthermore, it implements the cases
as 

if (expr == case1expr) {statements1}
else if (expr == case2expr) {statements2}
...

This means that when two cases have the same value, the second one will
be quietly ignored.

We have two options I think:

1. Remove the else. Run statements in all cases that have the right
value
2. Restrict case expressions to literals and check that there are no two
cases with the same value at compile-time.

Thoughts?

Mihael

_______________________________________________
Swift-devel mailing list
Swift-devel at ci.uchicago.edu
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20150529/338497ab/attachment.html>