From wilde at mcs.anl.gov  Wed May  1 10:47:05 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 1 May 2013 10:47:05 -0500 (CDT)
Subject: [Swift-devel] Shell through coasters
In-Reply-To: <1031339640.385488.1367392306983.JavaMail.root@ci.uchicago.edu>
Message-ID: <1797716937.6255151.1367423225902.JavaMail.root@mcs.anl.gov>


Mihael, a heads-up that we're trying to get this working.

David is investigating, but please provide any advice that may help get this working and available for general use.

Thanks,

- Mike

----- Forwarded Message -----
From: "David Kelly" <davidk at ci.uchicago.edu>
To: "Michael Wilde" <wilde at mcs.anl.gov>, "Ketan Maheshwari" <ketan at mcs.anl.gov>
Sent: Wednesday, May 1, 2013 2:11:46 AM
Subject: Shell through coasters

It doesn't work. But if it did work, this is how you would do it. 

1. Start swift with -tui 
2. F4 to go to list of jobs. 
3. Press enter on an active job 
4. A window pops up with job properties. Hit tab and select terminal. Then you'll see something like this: 

/000000$ ls 
java.lang.NullPointerException 
/000000$ pwd 
java.lang.NullPointerException 

I'll take a look at what's causing that - just wanted to show how it's supposed to work, in theory. 

David 


From wilde at mcs.anl.gov  Fri May  3 10:46:20 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 3 May 2013 10:46:20 -0500 (CDT)
Subject: [Swift-devel] Where is config info for SGE?
Message-ID: <914271405.6738462.1367595980630.JavaMail.root@mcs.anl.gov>


Im looking for config info on sites.xml for SGE clusters.  In particular where to specify the "pe".

Can anyone point me to this?

David, can you add it to the Site guide?

Thanks,

- Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From wilde at mcs.anl.gov  Fri May  3 10:49:13 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 3 May 2013 10:49:13 -0500 (CDT)
Subject: [Swift-devel] Where is config info for SGE?
In-Reply-To: <914271405.6738462.1367595980630.JavaMail.root@mcs.anl.gov>
Message-ID: <136990097.6738710.1367596153595.JavaMail.root@mcs.anl.gov>

OK, I found this in the email archives:

  <profile key="pe" namespace="globus">16way</profile>


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, May 3, 2013 10:46:20 AM
> Subject: [Swift-devel] Where is config info for SGE?
> 
> 
> Im looking for config info on sites.xml for SGE clusters.  In
> particular where to specify the "pe".
> 
> Can anyone point me to this?
> 
> David, can you add it to the Site guide?
> 
> Thanks,
> 
> - Mike
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wilde at mcs.anl.gov  Tue May  7 16:40:52 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 7 May 2013 16:40:52 -0500 (CDT)
Subject: [Swift-devel] Are automatic coaster timeout values adjustable?
Message-ID: <1830940329.1870183.1367962852834.JavaMail.root@mcs.anl.gov>


Can we set the timeouts for automatic coasters on say a per-site or per-run basis?

Specifically I may need this for persistent automatic coasters, to support an app that calls swift repeatedly to run an ensemble of MPI jobs. We'll want to ensure that the coaster workers dont shut down between swift runs, which may take a minute or so for the higher level script to generate after getting each round of results.

Thanks,

- Mike


From hategan at mcs.anl.gov  Tue May  7 18:55:21 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 07 May 2013 16:55:21 -0700
Subject: [Swift-devel] Are automatic coaster timeout values adjustable?
In-Reply-To: <1830940329.1870183.1367962852834.JavaMail.root@mcs.anl.gov>
References: <1830940329.1870183.1367962852834.JavaMail.root@mcs.anl.gov>
Message-ID: <1367970921.28903.8.camel@echo>

Not at this time. It can be added.

Mihael

On Tue, 2013-05-07 at 16:40 -0500, Michael Wilde wrote:
> Can we set the timeouts for automatic coasters on say a per-site or per-run basis?
> 
> Specifically I may need this for persistent automatic coasters, to support an app that calls swift repeatedly to run an ensemble of MPI jobs. We'll want to ensure that the coaster workers dont shut down between swift runs, which may take a minute or so for the higher level script to generate after getting each round of results.
> 
> Thanks,
> 
> - Mike
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From davidk at ci.uchicago.edu  Wed May  8 10:53:49 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Wed, 8 May 2013 10:53:49 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <1809849012.757125.1368023019921.JavaMail.root@ci.uchicago.edu>
Message-ID: <448164052.790944.1368028429410.JavaMail.root@ci.uchicago.edu>

Hello, 


Just a few random questions about wrapper staging: 


How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I have wrapper staging enabled, but Swift is trying to connect to the server and retrieve the file, rather than just passing the URI to the wrapper script. 


What does the property wrapper.staging.local.server do? 


>From within _swiftwrap.wrapperstaging, how can I know the current working directory of the user? I think I need this information to handle relative path names. 


I believe in trunk, filenames passed as arguments to Swift apps being prepended with "__root__". What does this mean, and should swiftwrap strip this out? 


Thanks, 
David 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130508/c8dab755/attachment.html>

From hategan at mcs.anl.gov  Wed May  8 12:46:45 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 08 May 2013 10:46:45 -0700
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <448164052.790944.1368028429410.JavaMail.root@ci.uchicago.edu>
References: <448164052.790944.1368028429410.JavaMail.root@ci.uchicago.edu>
Message-ID: <1368035205.7010.12.camel@echo>

I'm a bit busy today and this week, so I might not get to fixing these
until the weekend, but I'll try to answer some of your questions.

On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> Hello, 
> 
> 
> Just a few random questions about wrapper staging: 
> 
> 
> How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I
> have wrapper staging enabled, but Swift is trying to connect to the
> server and retrieve the file, rather than just passing the URI to the
> wrapper script. 

With wrapper staging enabled, swift should not try to *transfer* files,
but mappers might connect to the server to figure out what files are
there (i.e. to complete calls to Mapper.existing()).

> 
> What does the property wrapper.staging.local.server do? 

It specifies what url prefix to pass to the wrapper for local files
(i.e. accessible directly by swift). 

> 
> 
> From within _swiftwrap.wrapperstaging, how can I know the current
> working directory of the user? I think I need this information to
> handle relative path names. 

The wrapper should get a -urlprefix argument containing the CWD. The
value of that arg is constructed as: ${wrapper.staging.local.server} +
${CWD}.

> 
> 
> I believe in trunk, filenames passed as arguments to Swift apps being
> prepended with "__root__". What does this mean, and should swiftwrap
> strip this out? 

No. There is a mapping from local file names to remote file names. It
used to be that absolute files are mapped simply by removing the leading
slash. So "/dir/file" on the swift side would become "dir/file" on the
remote side. But then if there was a local "dir/file", it would also be
"dir/file" on the remote side, causing a conflict.

At some point in trunk, I changed that (and also the handling of "/../")
in response to a bug that, I believe, Ketan submitted. The change
involved mapping a leading slash to "__root__" and parent dir references
to "__parent__". They have no special meaning as far as the wrapper or
the application is concerned. They get the proper path through @filename
and the likes, and swift stages the files in the places pointed by
@filename.

Mihael



From davidk at ci.uchicago.edu  Wed May  8 14:11:06 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Wed, 8 May 2013 14:11:06 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <1368035205.7010.12.camel@echo>
Message-ID: <799981667.860037.1368040266667.JavaMail.root@ci.uchicago.edu>

Mihael, 

When I was testing with gsiftp I was using a dummy url, expecting the string to just get passed to the wrapper. I was confused when I saw a failed connection, but this makes sense now. 

Thanks for clarifying things. I should have enough info for now to keep pushing ahead on this. 

David 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130508/253815d7/attachment.html>

From dsk at ci.uchicago.edu  Thu May  9 11:44:57 2013
From: dsk at ci.uchicago.edu (Daniel S. Katz)
Date: Thu, 9 May 2013 12:44:57 -0400
Subject: [Swift-devel] Fwd: Google Groups Invitation: SHIWA User Forum
References: <518bceb4.0635310a.3f1f.3b64.GMR@mx.google.com>
Message-ID: <ED544F83-F70E-4878-B544-25E92A90C9BB@ci.uchicago.edu>

FYI...

https://groups.google.com/forum/#!forum/shiwa-user-forum

SHIWA is http://www.shiwa-workflow.eu - An EU project to make workflows interoperable, led by Peter Kacsuk

Dan



Begin forwarded message:

> From: "kittka (Google Groups)" <noreply at googlegroups.com>
> Subject: Google Groups Invitation: SHIWA User Forum
> Date: May 9, 2013 12:28:36 PM EDT
> To: d.katz at ieee.org
> Reply-To: vargaki at gmail.com
> 
> kittka vargaki at gmail.com has invited you to join the SHIWA User Forum group 
> with this message:
> 
> Dear SHIWA User Forum member,
> 
> I would like to kindly invite you to the new SHIWA User Forum. We moved the 
> User Forum to Google groups to make it easier to follow and comment on topics 
> of interest.
> 
> Please accept this invitation (...) to the new SHIWA
> User Forum that will be used from now on to discuss workflow sharing and 
> interoperability and user experience of the SHIWA Simulation Platform.
> 
> If you require any more information, please do not hesitate to contact me at 
> kitti.varga at sztaki.mta.hu.
> 
> Many thanks,
> 
> Kitti Varga
> on behalf of the SHIWA and ER-flow projects
> 
> Here is the group's description:
> 
> The SHIWA User Forum's goal is to build a powerful user community to discuss 
> the SHIWA technology, workflow sharing and interoperability.
> 


-- 
Daniel S. Katz
University of Chicago
(773) 834-7186 (voice)
(773) 834-6818 (fax)
d.katz at ieee.org or dsk at ci.uchicago.edu
http://www.ci.uchicago.edu/~dsk/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130509/dca8f53c/attachment.html>

From iraicu at cs.iit.edu  Thu May  9 17:03:51 2013
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Thu, 09 May 2013 17:03:51 -0500
Subject: [Swift-devel] Call for Participation - ACM HPDC 2013
Message-ID: <518C1D47.9030607@cs.iit.edu>


                  Call for Participation - HPDC-2013
                    New York City, NY, June 17-21

          Early registration is open! http://www.hpdc.org/2013

The ACM International Symposium on High-Performance Parallel and 
Distributed
Computing (HPDC) is the premier annual conference for presenting the latest
research on the design, implementation, evaluation, and the use of parallel
and distributed systems for high-end computing.

The 22nd HPDC conference and its seven associated workshops will take place
at the New Yorker Hotel June 17-21, featuring a technical program covering
various aspects of high-performance computing, three distinguished keynote
speakers, and an industry session:

Keynote speakers:
   Garth Gibson
     Professor of Computer Science at Carnegie Mellon University
     Co-founder of Panasas, Inc.
   David Shaw
     Chief scientist of D. E. Shaw Research
     Senior research fellow, Center for Computational Biology and
      Bioinformatics at Columbia University.
   Miron Livny
     Professor of Computer Sciences at University of Wisconsin-Madison
     Leader of the HTCondor project

Technical Sessions:
   I/O and Data-Intensive Computing
   Networks
   Communication
   Checkpointing, Bugs and Errors
   Multicore and GPUs
   Virtualization and Clouds

Industry session:
   Oscar Boykin - Twitter
   Blake G. Fitch - IBM Research
   (More to come..)

Workshops:
   EEHPDC (energy)
   FTXS (fault tolerance)
   CLHS (HPC security)
   HPPN (networking)
   OrmaCloud (cloud computing)
   ScienceCloud (cloud computing)
   VTDC (virtualization)

HPDC'13 Student Travel Grants Application
Applications due by May 31st, 2013
http://www.hpdc.org/2013/registration/student-travel-grants/

General Co-Chairs
   Manish Parashar - Rutgers University
   Jon Weissman - University of Minnesota
Program Co-Chairs
   Dick Epema - TU Delft
   Renato Figueiredo - University of Florida
Posters Chair
   Ivan Rodero - Rutgers University
Workshops Chair
   Abhishek Chandra - University of Minnesota

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud, IEEE/ACM DataCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================



From eskogen at g.clemson.edu  Fri May 10 09:03:27 2013
From: eskogen at g.clemson.edu (Eric Skogen)
Date: Fri, 10 May 2013 10:03:27 -0400
Subject: [Swift-devel] Revised Quickstart
Message-ID: <CAFFGBnTx96FvdCciG+zyqe6Zx5UpcO_dUQ4ryvY4aVfCcbGQrA@mail.gmail.com>

Attached is a revised Quickstart guide.  Building it requires that the
UofC_2013-04-09 folder be moved to the examples folder and renamed
"quickstart"

We hope this will help new users become comfortable with swift more quickly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130510/344b876a/attachment.html>
-------------- next part --------------
Swift Quickstart
================

[abstract]
Abstract
--------
This guide describes the steps needed to download, install, configure, 
and run the basic examples for Swift. If you are using a pre-installed 
version of Swift, you can skip directly to the configuration section.

Stable Releases vs. Development Releases
----------------------------------------
Stable releases of Swift have undergone more extensive testing than development releases.
In general, they are more stable, have fewer bugs, and have been tested on a variety of
systems.

The development version of Swift is aimed at developers and testers. The development 
code has the highest chance of containing buggy and untested code. If you need stability 
please use the latest stable release. 

Downloading a Swift Distribution
--------------------------------
There are two main ways of getting the Swift implementation: binary
releases and the source repository.

Binary Releases
~~~~~~~~~~~~~~~
For the majority of users, downloading and installing binary releases is recommended.
Since Swift is written in Java, the binary packages will run on all supported platforms with 
Java Runtime Environment 1.5 or greater. Binary releases can be obtained from the 
http://www.ci.uchicago.edu/swift/downloads/index.php[Swift downloads page]. 

Once downloaded, simply unpack the downloaded package (swift-<version>.tar.gz) into a
directory of your choice:

-----
tar -xzvf swift-<version>.tar.gz
-----

This will create a swift-<version> directory containing the build.

Source Repository
~~~~~~~~~~~~~~~~~
The source code for Swift is available to developers who have an interest in contributing 
new features. To build Swift from source code, you will need http://ant.apache.org/[Apache Ant]
and http://www.oracle.com/technetwork/java/javase/downloads/index.html[Java JDK]. Once
built, the dist/swift-svn directory will contain your build.

To download and build Swift 0.93, follow these instructions:
-----
$ mkdir swift-0.93 
$ cd swift-0.93 
$ svn co https://cogkit.svn.sourceforge.net/svnroot/cogkit/branches/4.1.9/src/cog 
$ cd cog/modules 
$ svn co https://svn.ci.uchicago.edu/svn/vdl2/branches/release-0.93 swift 
$ cd swift 
$ ant redist
-----

Setting your PATH
-----------------
Once Swift is installed, it is useful to add the swift binary to your PATH
environment variable. To do this, first determine where the Swift bin
directory is located. If you installed Swift from a binary release, it will
be in the swift-0.93/bin directory where you installed it. If you followed
the instructions above for installing Swift from a source repository, it
will be located in swift-0.93/cog/modules/swift/dist/swift-svn/bin.

Add the following line to the bottom of ~/.bashrc:

-----
export PATH=$PATH:/full/path/to/swift
-----

When you login, test this out by typing the command
-----
$ which swift
-----

This should point you to the path of the Swift binary.

Running Swift Examples
----------------------

This portion is intended to help you get started with Swift. You will use swift to generate data, both as named files on disc and as intermediate data and then pass it on to analytical tools. All the programs are deliberately simple because it is intended to show the Swift usage, not the analytical tool usage. Work the examples with the shell scripts then replace them with whatever analytical tool you want to use.

Change directory into the created directory called "tutorial" 
This directory contains: 
	bin: script tools for the tutorial 
	scripts: Swift scripts for the tutorial 

Start: 

-----
$ source setup.sh # to add tutorial programs to your PATH ...and then verify: 
$ random.sh # should be found in your PATH now The tutorial is arranged in parts. 
-----

Begin:

----- 
$ cd part01 
$ cat README
----- 
When finished: $ cd ../part02 # etc 

In each part, you can type "cleanup" after running a few swift scripts to remove the old logs that build up, and (usually) the output files. 

This tutorial takes you through a six step process that demonstrates how to effectively use Swift. It also demonstrates running it on a cluster. Naturally, configuration details will vary more in the demonstration than they did in the six step process.

.p1.swift
**************** 

----
include::../../examples/quickstart/part01/p1.swift[]
----
****************

The first part is a basic swift program that calls a shell script random.sh . Notice no arguments were passed to the app, but we did use the return value. The return value came from being on the left hand side of the = when the app was called.

Notice that although the output file is stored, no filename is given. The file is stored on disc and could be retrieved (it is in the _concurrent folder), but making the file anonymous (by doing what we did -- not giving it a name) should be done when the file is intended as intermediate data that will not be needed later. This next script shows how to name files. Not only are they easier to find, it is the appropriate style for files that are intended to persist outside the script.

Now that we can choose our greeting text, we can call the same procedure
with different parameters to generate several output files with
different greetings. The code is in manyparam.swift and can be run as
before using the swift command.

.p2b.swift
**************** 

----
include::../../examples/quickstart/part02/p2b.swift[]
----
****************

When you look at this file you can see that the output file is specified. In this case it will be named sim.out in folder output. Technically, it doesn?t matter if you name files or not since you can find anonymous ones and named ones can be ignored, but searching for files you want and filtering files you don?t can impact productivity more than may be initially apparent. 

.p3.swift
**************** 

----
include::../../examples/quickstart/part03/p3.swift[]
----
****************

Here we illustrate the use of a loop to run things concurrently. Notice that we do not specify anything about how many instances are running or where they are running. This isn?t terribly important as we are running everything locally, but this may be extremely important as we begin to run on more resources. 

.p4.swift
**************** 

----
include::../../examples/quickstart/part04/p4.swift[]
----
****************

The noteworthy part here is that each output file is named. This is done by inserting a variable (i in this case) into each file to make sure each haves a unique name. Although good, this pattern can be modified as long as something must be done to make each name unique.

.p5.swift
**************** 

----
include::../../examples/quickstart/part05/p5.swift[]
----
****************

This illustrates passing the output data on for additional analysis. This technique of specifying the next step can be repeated until the entire workflow is specified. 

.p6.swift
**************** 

----
include::../../examples/quickstart/part06/p6.swift[]
----
****************

This demonstrates a more complicated analysis program. Obviously, ?complicated? is a relative term here, but the technique is the same whether we?re passing one parameter to a simple program or several to a heavyweight analytical tool.

.p7.swift
**************** 

----
include::../../examples/quickstart/part07/p7.swift[]
----
****************

.tc
**************** 

----
include::../../examples/quickstart/part07/tc[]
----
****************

.sites.xml
**************** 

----
include::../../examples/quickstart/part07/sites.xml[]
----
****************

This shows the changes necessary to run this script on a cluster. The specific configuration will probably not be directly useable unless you have access to this particular cluster, but it should give you an idea of what information is needed and what format it is expected in.

Notice that none of the changes in the .swift file actually have anything to do with running on a cluster. The additional parameters are there to allow us to better see performance changes, not running on a server. The changes that allow us to run on the cluster are in the tc and sites files.

We've now created a simple workflow with Swift. You can add programs to a workflow and manage their output. To make them more complicate you need only call more advanced utilities and repeat the techniques you have learned. There are a lot more things you can learn to make Swift work better, and you will find those in the user manual. Go ahead and experiment with it a little.

More documentation on how to run Swift can be found at 
http://www.ci.uchicago.edu/swift/docs/index.php.

From davidk at ci.uchicago.edu  Tue May 14 13:24:24 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Tue, 14 May 2013 13:24:24 -0500 (CDT)
Subject: [Swift-devel] Code style
In-Reply-To: <733861746.1635356.1368554459579.JavaMail.root@ci.uchicago.edu>
Message-ID: <1749735482.1644512.1368555864175.JavaMail.root@ci.uchicago.edu>


Hello, 


The topic of code style for internal Swift code came up this morning. On the swift-devel page, there is a document that describes how code should be formatted: https://sites.google.com/site/swiftdevel/internals/code-style. I think Justin wrote this document? I'm not sure we've ever discussed it much. Do people agree with these suggestions? Should we adopt this as our standard? Is it worth it go back to old code and reformat it? 


Another option may be something like the Oracle Java coding standard: http://www.oracle.com/technetwork/java/codeconvtoc-136057.html. 


David 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130514/28a0af42/attachment.html>

From wilde at mcs.anl.gov  Tue May 14 13:33:04 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 14 May 2013 13:33:04 -0500 (CDT)
Subject: [Swift-devel] Code style
In-Reply-To: <1749735482.1644512.1368555864175.JavaMail.root@ci.uchicago.edu>
Message-ID: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>


The internals/code-style page mentioned below is rather vague.

Im in favor of 3 things:

- code format inspired by K&R (more compact than {} on separate lines, while quite readable). The Oracle page cited by David is fine in that regard.

- spaces instead of hard tabs. I feel adamant about this.

- Ideally, a format for which some auto-formatter exists that would enable us to clean up our source.

Somehow the PBS provider files got very badly un-tabbed, so they are hard to read. We could at least fix the most offensive files. Does anyone have ideas on how to format all files without a massive SVN revision bump?  Is a full sweep a bad idea?

- Mike

----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Tuesday, May 14, 2013 1:24:24 PM
> Subject: [Swift-devel] Code style
> 
> 
> 
> 
> Hello,
> 
> 
> The topic of code style for internal Swift code came up this morning.
> On the swift-devel page, there is a document that describes how code
> should be formatted:
> https://sites.google.com/site/swiftdevel/internals/code-style. I
> think Justin wrote this document? I'm not sure we've ever discussed
> it much. Do people agree with these suggestions? Should we adopt
> this as our standard? Is it worth it go back to old code and
> reformat it?
> 
> 
> Another option may be something like the Oracle Java coding standard:
> http://www.oracle.com/technetwork/java/codeconvtoc-136057.html.
> 
> 
> David
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From benc at hawaga.org.uk  Tue May 14 13:48:52 2013
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 14 May 2013 18:48:52 +0000 (UTC)
Subject: [Swift-devel] Code style
In-Reply-To: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
Message-ID: <alpine.DEB.2.00.1305141846410.30194@dildano.hawaga.org.uk>

> Somehow the PBS provider files got very badly un-tabbed, so they are 
> hard to read. We could at least fix the most offensive files. Does 
> anyone have ideas on how to format all files without a massive SVN 
> revision bump?  Is a full sweep a bad idea?

You basically can't do it in a nice way. Its a fairly compelling argument 
against bulk reformatting if you value line-by-line last-change history 
(which I find myself valuing less and less in other projects).

-- 


From hategan at mcs.anl.gov  Tue May 14 13:50:57 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 11:50:57 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <1749735482.1644512.1368555864175.JavaMail.root@ci.uchicago.edu>
References: <1749735482.1644512.1368555864175.JavaMail.root@ci.uchicago.edu>
Message-ID: <1368557457.25028.16.camel@echo>

On Tue, 2013-05-14 at 13:24 -0500, David Kelly wrote:
> Hello, 
> 
> 
> The topic of code style for internal Swift code came up this morning.
> On the swift-devel page, there is a document that describes how code
> should be formatted:
> https://sites.google.com/site/swiftdevel/internals/code-style. I think
> Justin wrote this document? I'm not sure we've ever discussed it much.
> Do people agree with these suggestions? Should we adopt this as our
> standard? Is it worth it go back to old code and reformat it? 

Mostly. I don't much agree with the 70 character lines. I think the age
of 80 column terminals has been gone for a while, and that limit tends
to make code unreadable since logical units span too many lines and are
split in the weirdest of places.

So I'd vote for something more reasonable, such as 110.

I'd like to add (Oracle also has most of these):
- always using brackets (even for single-instruction then/else). It
delimits the scope of conditions and allows for easily adding things to
branches. 
- use spaces before and after operators (e.g. 3 + 2 instead of 3+2).
- else, catch on next line. Most of the code is like that already. e.g.:
   if (condition) {
   }
   else {
   }
- sufficiently indent subsequent parts of a line-split thing to indicate
subordination to the beginning of the thing as well as distinguish from
other sub-ordinate things. Example
   private void method(int x)
           throws Exception {
       // code
   }
  In the above example it is very quickly clear that the throws clause
belongs to method and that it's not part of the code. 

   The not so good choices (IMO) are:

   private void method(int x)
   throws Exception {
       // code
   }
   (hard to tell without reading that the throws clause is not a method
since it's on the same level as method declarations)

  private void method(int x)
       throws Exception {
       // code
   }
   (hard to tell the throws clause is not part of the code unless you
scan to the end of the line and spot the "{")
- do indent cases in a switch statement.

Mihael

> 
> 
> Another option may be something like the Oracle Java coding standard: http://www.oracle.com/technetwork/java/codeconvtoc-136057.html. 
> 
> 
> David 
> 
> Hello,
> 
> 
> The topic of code style for internal Swift code came up this morning.
> On the swift-devel page, there is a document that describes how code
> should be formatted:
> https://sites.google.com/site/swiftdevel/internals/code-style. I think
> Justin wrote this document? I'm not sure we've ever discussed it much.
> Do people agree with these suggestions? Should we adopt this as our
> standard? Is it worth it go back to old code and reformat it?
> 
> 
> Another option may be something like the Oracle Java coding
> standard: http://www.oracle.com/technetwork/java/codeconvtoc-136057.html.
> 
> 
> David
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel




From hategan at mcs.anl.gov  Tue May 14 14:26:49 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 12:26:49 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
Message-ID: <1368559609.26908.7.camel@echo>

On Tue, 2013-05-14 at 13:33 -0500, Michael Wilde wrote:
> The internals/code-style page mentioned below is rather vague.
> 
> Im in favor of 3 things:
> 
> - code format inspired by K&R (more compact than {} on separate lines, while quite readable). The Oracle page cited by David is fine in that regard.

I don't have a good reason for the opposite. I think I prefer the
aesthetics of being similar to <letters> <brackets> which is mandated by
method/class definitions and pretty much everything else. Also, most of
the code already uses this, and I'd prefer that for consistency reasons.

> 
> - spaces instead of hard tabs. I feel adamant about this.

Eclipse has some options where you can do both "keep as is" (for
existing lines) and "use spaces for new stuff".

> 
> - Ideally, a format for which some auto-formatter exists that would enable us to clean up our source.
> 
> Somehow the PBS provider files got very badly un-tabbed, so they are
> hard to read.

I think an easy solution is setting the tab size to 4 in your favorite
editor and that makes the problem magically go away.

Mihael



From benc at hawaga.org.uk  Tue May 14 14:39:16 2013
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 14 May 2013 19:39:16 +0000 (UTC)
Subject: [Swift-devel] Code style
In-Reply-To: <1368559609.26908.7.camel@echo>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
Message-ID: <alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>


> > - spaces instead of hard tabs. I feel adamant about this.
> 
> Eclipse has some options where you can do both "keep as is" (for
> existing lines) and "use spaces for new stuff".

both of those at the same time is often terrible because of the ambiguity 
in how long a tab is. if you have some formatted code and you want to add 
lines to it, pretty much all you can do is use the same byte sequence for 
indents as has already been used to indent that same block, and then add 
additional whitespace on the end to get further depth.

(mmm tabs vs spaces debate)

-- 



From hategan at mcs.anl.gov  Tue May 14 14:43:24 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 12:43:24 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
	<alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
Message-ID: <1368560604.26908.9.camel@echo>

On Tue, 2013-05-14 at 19:39 +0000, Ben Clifford wrote:
> 
> > > - spaces instead of hard tabs. I feel adamant about this.
> > 
> > Eclipse has some options where you can do both "keep as is" (for
> > existing lines) and "use spaces for new stuff".
> 
> both of those at the same time is often terrible because of the ambiguity 
> in how long a tab is.

There is an option for that, and there you can say 1 tab = 4 spaces.

>  if you have some formatted code and you want to add 
> lines to it, pretty much all you can do is use the same byte sequence for 
> indents as has already been used to indent that same block, and then add 
> additional whitespace on the end to get further depth.
> 
> (mmm tabs vs spaces debate)
> 
> -- 
> 




From benc at hawaga.org.uk  Tue May 14 14:47:49 2013
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 14 May 2013 19:47:49 +0000 (UTC)
Subject: [Swift-devel] Code style
In-Reply-To: <1368560604.26908.9.camel@echo>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
	<alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
	<1368560604.26908.9.camel@echo>
Message-ID: <alpine.DEB.2.00.1305141944310.26550@dildano.hawaga.org.uk>


> > both of those at the same time is often terrible because of the ambiguity 
> > in how long a tab is.
> 
> There is an option for that, and there you can say 1 tab = 4 spaces.

But a tab isn't 4 spaces.

If someone wrote some code with tab = 4, and committed it, and someone 
else wrote code with tab = 8, maybe even in the same file, there is no 
consistent value to use.

The only thing you can consistently do is copy the byte sequence that was 
used for the block you're adding to. (and even that only works if the 
indentation in that block is already self-consistent...)

Or ban tabs and reformat them away and dock 1 beer point from each person 
who committed the tabs in the first place...

-- 



From hategan at mcs.anl.gov  Tue May 14 15:13:16 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 13:13:16 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <alpine.DEB.2.00.1305141944310.26550@dildano.hawaga.org.uk>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
	<alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
	<1368560604.26908.9.camel@echo>
	<alpine.DEB.2.00.1305141944310.26550@dildano.hawaga.org.uk>
Message-ID: <1368562396.26908.32.camel@echo>

On Tue, 2013-05-14 at 19:47 +0000, Ben Clifford wrote:
> 
> > > both of those at the same time is often terrible because of the ambiguity 
> > > in how long a tab is.
> > 
> > There is an option for that, and there you can say 1 tab = 4 spaces.
> 
> But a tab isn't 4 spaces.

A tab is an indentation unit. How big you want to see that on your
screen is another issue.

> 
> If someone wrote some code with tab = 4, and committed it, and someone 
> else wrote code with tab = 8, maybe even in the same file, there is no 
> consistent value to use.

A tab is a tab. You can't write a file with both 8 space tabs and 4
space tabs. You can however write a file with 8 space indentations vs. 4
space indentations if you don't use tabs exclusively.

> 
> The only thing you can consistently do is copy the byte sequence that was 
> used for the block you're adding to. (and even that only works if the 
> indentation in that block is already self-consistent...)

That's what eclipse does I believe. But that obviously doesn't apply for
things with no existing indentation, like new files, or when you hit
enter, delete all leading whitespace, and then start a new method.

> 
> Or ban tabs and reformat them away and dock 1 beer point from each person 
> who committed the tabs in the first place...

I did, and you are not getting beer from me. Tabs are clearly superior.
I should get the beer!

:)

Anyway, I think we should drop this. The mess goes away if you set your
tab size appropriately. This was never a serious issue for me
(regardless what files I looked at - if they mix tab and spaces, you can
infer the tab size and adjust accordingly). It's certainly easier than
having this discussion, and it's a one-time operation if you use any
decent IDE/editor that can save per-project settings. And if you don't
use a decent IDE it probably means you are not spending much time on
this, so you could be nice and follow the existing conventions when
making a change. And if you are not nice, that's ok too. We'll probably
re-format your code when we see it, but please don't complain that the
whole project should be changed because your patch uses a different
indentation scheme.

As for new files, I don't care, but if you use spaces, please use 4 of
them for indentation. 2 is too little visually, and 8 is a waste of
horizontal space. I won't go into the odd numbers.

Mihael




From benc at hawaga.org.uk  Tue May 14 15:28:02 2013
From: benc at hawaga.org.uk (Ben Clifford)
Date: Tue, 14 May 2013 20:28:02 +0000 (UTC)
Subject: [Swift-devel] Code style
In-Reply-To: <1368562396.26908.32.camel@echo>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
	<alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
	<1368560604.26908.9.camel@echo>
	<alpine.DEB.2.00.1305141944310.26550@dildano.hawaga.org.uk>
	<1368562396.26908.32.camel@echo>
Message-ID: <alpine.DEB.2.00.1305142024460.26550@dildano.hawaga.org.uk>


oh i can argue this all night.

> > If someone wrote some code with tab = 4, and committed it, and someone 
> > else wrote code with tab = 8, maybe even in the same file, there is no 
> > consistent value to use.
> 
> A tab is a tab. You can't write a file with both 8 space tabs and 4
> space tabs. You can however write a file with 8 space indentations vs. 4
> space indentations if you don't use tabs exclusively.

Person A commits some code with a single tab indent.
Person B has their display set to tab = 8 and adds new lines with 8 spaces 
rather than using the existing indentation byte sequence.
Person C has their display set to tab = 4 and adds new lines elsewhere in 
the file with 4 spaces, rather than using the existing indentation byte 
sequence.

Now you have a file with both 8 space tabs and 4 space tabs.

it seems hardest of all to stop people using their own favourite 
indentation sequence rather than the pattern already used. (make tabs and 
spaces have different colours and a bunch of the communally edited code I 
work with looks pretty puke)

-- 



From tim.g.armstrong at gmail.com  Tue May 14 15:35:21 2013
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Tue, 14 May 2013 22:35:21 +0200
Subject: [Swift-devel] Code style
In-Reply-To: <alpine.DEB.2.00.1305142024460.26550@dildano.hawaga.org.uk>
References: <589970138.1693134.1368556384080.JavaMail.root@mcs.anl.gov>
	<1368559609.26908.7.camel@echo>
	<alpine.DEB.2.00.1305141937000.30194@dildano.hawaga.org.uk>
	<1368560604.26908.9.camel@echo>
	<alpine.DEB.2.00.1305141944310.26550@dildano.hawaga.org.uk>
	<1368562396.26908.32.camel@echo>
	<alpine.DEB.2.00.1305142024460.26550@dildano.hawaga.org.uk>
Message-ID: <CAC0jiV7MHnC1_AeAB=2wG_bjUq+r-fO_8Vg1nr5nSqRqVPuRpQ@mail.gmail.com>

 I knew this was going to end badly as soon as I saw the initial email.

2 spaces, no tabs, } else {, 80 characters wide.  Discuss.

The only helpful comment I have with the discussion is that it's a real
pain to maintain patch-sets (or a code branch) against code with
inconsistent indentation or formatting, since you can't fix any of it
without ruining your ability to merge changes.  This doesn't seem to be a
major problem for Swift.

- Tim


On Tue, May 14, 2013 at 10:28 PM, Ben Clifford <benc at hawaga.org.uk> wrote:

>
> oh i can argue this all night.
>
> > > If someone wrote some code with tab = 4, and committed it, and someone
> > > else wrote code with tab = 8, maybe even in the same file, there is no
> > > consistent value to use.
> >
> > A tab is a tab. You can't write a file with both 8 space tabs and 4
> > space tabs. You can however write a file with 8 space indentations vs. 4
> > space indentations if you don't use tabs exclusively.
>
> Person A commits some code with a single tab indent.
> Person B has their display set to tab = 8 and adds new lines with 8 spaces
> rather than using the existing indentation byte sequence.
> Person C has their display set to tab = 4 and adds new lines elsewhere in
> the file with 4 spaces, rather than using the existing indentation byte
> sequence.
>
> Now you have a file with both 8 space tabs and 4 space tabs.
>
> it seems hardest of all to stop people using their own favourite
> indentation sequence rather than the pattern already used. (make tabs and
> spaces have different colours and a bunch of the communally edited code I
> work with looks pretty puke)
>
> --
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130514/9984d912/attachment.html>

From wilde at mcs.anl.gov  Tue May 14 15:52:21 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 14 May 2013 15:52:21 -0500 (CDT)
Subject: [Swift-devel] Code style
In-Reply-To: <CAC0jiV7MHnC1_AeAB=2wG_bjUq+r-fO_8Vg1nr5nSqRqVPuRpQ@mail.gmail.com>
Message-ID: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>

> I knew this was going to end badly as soon as I saw the initial
> email.

Ending badly would be good, compared to not ending :)

After a few decades of not messing with emacs tab settings I set it to 4 spaces and the offending file views OK now.

I withdraw the suggestion to make any changes.  Lets just keep Swift source looking like it does now, thats fine.

- Mike


From wozniak at mcs.anl.gov  Tue May 14 22:57:24 2013
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Tue, 14 May 2013 22:57:24 -0500
Subject: [Swift-devel] Code style
In-Reply-To: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>
References: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>
Message-ID: <519307A4.1030603@mcs.anl.gov>

On 5/14/2013 3:52 PM, Michael Wilde wrote:
>> I knew this was going to end badly as soon as I saw the initial
>> email.
> Ending badly would be good, compared to not ending :)
>
> After a few decades of not messing with emacs tab settings I set it to 4 spaces and the offending file views OK now.
>
> I withdraw the suggestion to make any changes.  Lets just keep Swift source looking like it does now, thats fine.

I put the note about tabs in the document as a note to new developers 
and a concession to the fact it was used so much in the existing code.  
If anyone can add some notes on there about how to best get this working 
in the latest Eclipse and Emacs, that would help a lot (I don't think I 
ever got it working with Emacs).

Some nice points about style are in this article: 
http://queue.acm.org/detail.cfm?id=2063168



From hategan at mcs.anl.gov  Tue May 14 23:20:44 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 21:20:44 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <519307A4.1030603@mcs.anl.gov>
References: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>
	<519307A4.1030603@mcs.anl.gov>
Message-ID: <1368591644.2784.1.camel@echo>

On Tue, 2013-05-14 at 22:57 -0500, Justin M Wozniak wrote:
> On 5/14/2013 3:52 PM, Michael Wilde wrote:
> >> I knew this was going to end badly as soon as I saw the initial
> >> email.
> > Ending badly would be good, compared to not ending :)
> >
> > After a few decades of not messing with emacs tab settings I set it to 4 spaces and the offending file views OK now.
> >
> > I withdraw the suggestion to make any changes.  Lets just keep Swift source looking like it does now, thats fine.
> 
> I put the note about tabs in the document as a note to new developers 
> and a concession to the fact it was used so much in the existing code.  
> If anyone can add some notes on there about how to best get this working 
> in the latest Eclipse and Emacs, that would help a lot (I don't think I 
> ever got it working with Emacs).

What do you mean by "this"?

Mihael



From wozniak at mcs.anl.gov  Tue May 14 23:44:59 2013
From: wozniak at mcs.anl.gov (Justin M Wozniak)
Date: Tue, 14 May 2013 23:44:59 -0500
Subject: [Swift-devel] Code style
In-Reply-To: <1368591644.2784.1.camel@echo>
References: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>
	<519307A4.1030603@mcs.anl.gov> <1368591644.2784.1.camel@echo>
Message-ID: <519312CB.7020604@mcs.anl.gov>

On 05/14/2013 11:20 PM, Mihael Hategan wrote:
> On Tue, 2013-05-14 at 22:57 -0500, Justin M Wozniak wrote:
>>
>> I put the note about tabs in the document as a note to new developers
>> and a concession to the fact it was used so much in the existing code.
>> If anyone can add some notes on there about how to best get this working
>> in the latest Eclipse and Emacs, that would help a lot (I don't think I
>> ever got it working with Emacs).
> What do you mean by "this"?

The tabs/spaces thing.  Can the Eclipse settings be exported and 
posted?  (Now that I think about it, I use the Emacs key settings in 
Eclipse, with tab mapped to indent, which maybe made it harder for me to 
adapt to the tabs+spaces convention.  Maybe this is easier for others.)

-- 
Justin M Wozniak



From hategan at mcs.anl.gov  Wed May 15 00:09:20 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Tue, 14 May 2013 22:09:20 -0700
Subject: [Swift-devel] Code style
In-Reply-To: <519312CB.7020604@mcs.anl.gov>
References: <1248077145.1811931.1368564741029.JavaMail.root@mcs.anl.gov>
	<519307A4.1030603@mcs.anl.gov> <1368591644.2784.1.camel@echo>
	<519312CB.7020604@mcs.anl.gov>
Message-ID: <1368594560.3500.13.camel@echo>

On Tue, 2013-05-14 at 23:44 -0500, Justin M Wozniak wrote:
> On 05/14/2013 11:20 PM, Mihael Hategan wrote:
> > On Tue, 2013-05-14 at 22:57 -0500, Justin M Wozniak wrote:
> >>
> >> I put the note about tabs in the document as a note to new developers
> >> and a concession to the fact it was used so much in the existing code.
> >> If anyone can add some notes on there about how to best get this working
> >> in the latest Eclipse and Emacs, that would help a lot (I don't think I
> >> ever got it working with Emacs).
> > What do you mean by "this"?
> 
> The tabs/spaces thing.  Can the Eclipse settings be exported and 
> posted?

Actually, yes. Here's what I used in CoG (and what Swift inherited). It
uses spaces.

>   (Now that I think about it, I use the Emacs key settings in 
> Eclipse, with tab mapped to indent, which maybe made it harder for me to 
> adapt to the tabs+spaces convention.  Maybe this is easier for others.)
> 

These settings are for the formatter. By default, eclipse won't change
tabs into spaces or the other way around (unless you explicitly
re-format the code), but new things you write are affected by these
settings. For existing files it looks like it will copy existing
indentation if you hit enter, but then do its own when you indent
further.

So with spaces on and if you hit enter then tab on some existing line
that goes like

<TAB><TAB>if (i == 0) {

You get

<TAB><TAB>    //<-- you are here

Mihael

-------------- next part --------------
A non-text attachment was scrubbed...
Name: spaces2.xml
Type: application/xml
Size: 29585 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130514/3130f35c/attachment.xml>

From davidk at ci.uchicago.edu  Thu May 16 13:42:41 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Thu, 16 May 2013 13:42:41 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <1368035205.7010.12.camel@echo>
Message-ID: <693725412.2475823.1368729761044.JavaMail.root@ci.uchicago.edu>

I'd like to find a way to define different staging methods on a file-by-file basis. In some cases it would make sense to use dd to copy a file to the work directory. In other situations it may be better to create a symlink pointing to what you need. 


I'm thinking of something along these lines (names will probably get changed): 


file data <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">; // Will stage the data to the work directory 


file data2 <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink 


Then based on the prefix you define, _swiftwrap.wrapperstaging can know exactly how to stage the file. 


Is it possible to create data provider aliases, making file-direct:// and file-indirect:// work exactly like file:// until it gets passed to the wrapper? 


Thanks, 
David 

----- Original Message -----


From: "Mihael Hategan" <hategan at mcs.anl.gov> 
To: "David Kelly" <davidk at ci.uchicago.edu> 
Cc: "Swift Devel" <swift-devel at ci.uchicago.edu> 
Sent: Wednesday, May 8, 2013 12:46:45 PM 
Subject: Re: Wrapper staging 

I'm a bit busy today and this week, so I might not get to fixing these 
until the weekend, but I'll try to answer some of your questions. 

On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote: 
> Hello, 
> 
> 
> Just a few random questions about wrapper staging: 
> 
> 
> How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I 
> have wrapper staging enabled, but Swift is trying to connect to the 
> server and retrieve the file, rather than just passing the URI to the 
> wrapper script. 

With wrapper staging enabled, swift should not try to *transfer* files, 
but mappers might connect to the server to figure out what files are 
there (i.e. to complete calls to Mapper.existing()). 

> 
> What does the property wrapper.staging.local.server do? 

It specifies what url prefix to pass to the wrapper for local files 
(i.e. accessible directly by swift). 

> 
> 
> From within _swiftwrap.wrapperstaging, how can I know the current 
> working directory of the user? I think I need this information to 
> handle relative path names. 

The wrapper should get a -urlprefix argument containing the CWD. The 
value of that arg is constructed as: ${wrapper.staging.local.server} + 
${CWD}. 

> 
> 
> I believe in trunk, filenames passed as arguments to Swift apps being 
> prepended with "__root__". What does this mean, and should swiftwrap 
> strip this out? 

No. There is a mapping from local file names to remote file names. It 
used to be that absolute files are mapped simply by removing the leading 
slash. So "/dir/file" on the swift side would become "dir/file" on the 
remote side. But then if there was a local "dir/file", it would also be 
"dir/file" on the remote side, causing a conflict. 

At some point in trunk, I changed that (and also the handling of "/../") 
in response to a bug that, I believe, Ketan submitted. The change 
involved mapping a leading slash to "__root__" and parent dir references 
to "__parent__". They have no special meaning as far as the wrapper or 
the application is concerned. They get the proper path through @filename 
and the likes, and swift stages the files in the places pointed by 
@filename. 

Mihael 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130516/8160e9c9/attachment.html>

From hategan at mcs.anl.gov  Thu May 16 13:53:07 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 16 May 2013 11:53:07 -0700
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <693725412.2475823.1368729761044.JavaMail.root@ci.uchicago.edu>
References: <693725412.2475823.1368729761044.JavaMail.root@ci.uchicago.edu>
Message-ID: <1368730387.23294.4.camel@echo>

This is beginning to sound awfully like CDM.

So I'm wondering if anybody remembers why we started this wrapper
staging business instead of using CDM.

To answer your question, yes. The wrapper staging code should just pass
the protocol to the wrapper, but mappers might have some trouble with
some things, so I might have to make some changes.

Mihael

On Thu, 2013-05-16 at 13:42 -0500, David Kelly wrote:
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need. 
> 
> 
> I'm thinking of something along these lines (names will probably get changed): 
> 
> 
> file data <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">; // Will stage the data to the work directory 
> 
> 
> file data2 <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink 
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can know exactly how to stage the file. 
> 
> 
> Is it possible to create data provider aliases, making file-direct:// and file-indirect:// work exactly like file:// until it gets passed to the wrapper? 
> 
> 
> Thanks, 
> David 
> 
> ----- Original Message -----
> 
> 
> From: "Mihael Hategan" <hategan at mcs.anl.gov> 
> To: "David Kelly" <davidk at ci.uchicago.edu> 
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu> 
> Sent: Wednesday, May 8, 2013 12:46:45 PM 
> Subject: Re: Wrapper staging 
> 
> I'm a bit busy today and this week, so I might not get to fixing these 
> until the weekend, but I'll try to answer some of your questions. 
> 
> On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote: 
> > Hello, 
> > 
> > 
> > Just a few random questions about wrapper staging: 
> > 
> > 
> > How can I make gsiftp:// URIs get sent to _swiftwrap.wrapperstaging? I 
> > have wrapper staging enabled, but Swift is trying to connect to the 
> > server and retrieve the file, rather than just passing the URI to the 
> > wrapper script. 
> 
> With wrapper staging enabled, swift should not try to *transfer* files, 
> but mappers might connect to the server to figure out what files are 
> there (i.e. to complete calls to Mapper.existing()). 
> 
> > 
> > What does the property wrapper.staging.local.server do? 
> 
> It specifies what url prefix to pass to the wrapper for local files 
> (i.e. accessible directly by swift). 
> 
> > 
> > 
> > From within _swiftwrap.wrapperstaging, how can I know the current 
> > working directory of the user? I think I need this information to 
> > handle relative path names. 
> 
> The wrapper should get a -urlprefix argument containing the CWD. The 
> value of that arg is constructed as: ${wrapper.staging.local.server} + 
> ${CWD}. 
> 
> > 
> > 
> > I believe in trunk, filenames passed as arguments to Swift apps being 
> > prepended with "__root__". What does this mean, and should swiftwrap 
> > strip this out? 
> 
> No. There is a mapping from local file names to remote file names. It 
> used to be that absolute files are mapped simply by removing the leading 
> slash. So "/dir/file" on the swift side would become "dir/file" on the 
> remote side. But then if there was a local "dir/file", it would also be 
> "dir/file" on the remote side, causing a conflict. 
> 
> At some point in trunk, I changed that (and also the handling of "/../") 
> in response to a bug that, I believe, Ketan submitted. The change 
> involved mapping a leading slash to "__root__" and parent dir references 
> to "__parent__". They have no special meaning as far as the wrapper or 
> the application is concerned. They get the proper path through @filename 
> and the likes, and swift stages the files in the places pointed by 
> @filename. 
> 
> Mihael 
> 
> 
> 
> 
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need.
> 
> 
> I'm thinking of something along these lines (names will probably get
> changed):
> 
> 
> file data
> <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
>      // Will stage the data to the work directory
> file data2
> <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">; // Will create a symlink
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can
> know exactly how to stage the file. 
> 
> 
> Is it possible to create data provider aliases, making file-direct://
> and file-indirect:// work exactly like file:// until it gets passed to
> the wrapper?
> 
> 
> Thanks,
> David
> 
> 
> ______________________________________________________________________
>         From: "Mihael Hategan" <hategan at mcs.anl.gov>
>         To: "David Kelly" <davidk at ci.uchicago.edu>
>         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>         Sent: Wednesday, May 8, 2013 12:46:45 PM
>         Subject: Re: Wrapper staging
>         
>         I'm a bit busy today and this week, so I might not get to
>         fixing these
>         until the weekend, but I'll try to answer some of your
>         questions.
>         
>         On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
>         > Hello, 
>         > 
>         > 
>         > Just a few random questions about wrapper staging: 
>         > 
>         > 
>         > How can I make gsiftp:// URIs get sent to
>         _swiftwrap.wrapperstaging? I
>         > have wrapper staging enabled, but Swift is trying to connect
>         to the
>         > server and retrieve the file, rather than just passing the
>         URI to the
>         > wrapper script. 
>         
>         With wrapper staging enabled, swift should not try to
>         *transfer* files,
>         but mappers might connect to the server to figure out what
>         files are
>         there (i.e. to complete calls to Mapper.existing()).
>         
>         > 
>         > What does the property wrapper.staging.local.server do? 
>         
>         It specifies what url prefix to pass to the wrapper for local
>         files
>         (i.e. accessible directly by swift). 
>         
>         > 
>         > 
>         > From within _swiftwrap.wrapperstaging, how can I know the
>         current
>         > working directory of the user? I think I need this
>         information to
>         > handle relative path names. 
>         
>         The wrapper should get a -urlprefix argument containing the
>         CWD. The
>         value of that arg is constructed as:
>         ${wrapper.staging.local.server} +
>         ${CWD}.
>         
>         > 
>         > 
>         > I believe in trunk, filenames passed as arguments to Swift
>         apps being
>         > prepended with "__root__". What does this mean, and should
>         swiftwrap
>         > strip this out? 
>         
>         No. There is a mapping from local file names to remote file
>         names. It
>         used to be that absolute files are mapped simply by removing
>         the leading
>         slash. So "/dir/file" on the swift side would become
>         "dir/file" on the
>         remote side. But then if there was a local "dir/file", it
>         would also be
>         "dir/file" on the remote side, causing a conflict.
>         
>         At some point in trunk, I changed that (and also the handling
>         of "/../")
>         in response to a bug that, I believe, Ketan submitted. The
>         change
>         involved mapping a leading slash to "__root__" and parent dir
>         references
>         to "__parent__". They have no special meaning as far as the
>         wrapper or
>         the application is concerned. They get the proper path through
>         @filename
>         and the likes, and swift stages the files in the places
>         pointed by
>         @filename.
>         
>         Mihael
>         
> 
> 




From wilde at mcs.anl.gov  Thu May 16 14:00:39 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 16 May 2013 14:00:39 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <1368730387.23294.4.camel@echo>
Message-ID: <87969379.2767252.1368730839899.JavaMail.root@mcs.anl.gov>


We started wrapper staging for the broader goal of revisiting and improving the way we do all kinds of data management.

An initial goal is to allow a combination of direct access and local copies based on much simpler defaults, and the ability to specify a simple default in swift.properties and override at the point of mapping.

Later goals include using Globus staging tools (globus-url-copy and Globus Online) instead of the outdated GridFTP provider based on JGlobus.

Further goals include using catalog-look for files and doing compute node file caching, and improving on the current CDM implementation.

We want to create data management modes that will work in both Swift1 (/K) and Swift2 (/T) and which are cleanly isolated from the core language implementation.

- Mike

----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, May 16, 2013 1:53:07 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> This is beginning to sound awfully like CDM.
> 
> So I'm wondering if anybody remembers why we started this wrapper
> staging business instead of using CDM.
> 
> To answer your question, yes. The wrapper staging code should just
> pass
> the protocol to the wrapper, but mappers might have some trouble with
> some things, so I might have to make some changes.
> 
> Mihael
> 
> On Thu, 2013-05-16 at 13:42 -0500, David Kelly wrote:
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> > 
> > 
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct:// and file-indirect:// work exactly like file://
> > until it gets passed to the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > ----- Original Message -----
> > 
> > 
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> > 
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> > 
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > > 
> > > 
> > > Just a few random questions about wrapper staging:
> > > 
> > > 
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> > 
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> > 
> > > 
> > > What does the property wrapper.staging.local.server do?
> > 
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> > 
> > > 
> > > 
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> > 
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server} +
> > ${CWD}.
> > 
> > > 
> > > 
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> > 
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also be
> > "dir/file" on the remote side, causing a conflict.
> > 
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> > 
> > Mihael
> > 
> > 
> > 
> > 
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> >      // Will stage the data to the work directory
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to
> > the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > 
> > ______________________________________________________________________
> >         From: "Mihael Hategan" <hategan at mcs.anl.gov>
> >         To: "David Kelly" <davidk at ci.uchicago.edu>
> >         Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> >         Sent: Wednesday, May 8, 2013 12:46:45 PM
> >         Subject: Re: Wrapper staging
> >         
> >         I'm a bit busy today and this week, so I might not get to
> >         fixing these
> >         until the weekend, but I'll try to answer some of your
> >         questions.
> >         
> >         On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> >         > Hello,
> >         > 
> >         > 
> >         > Just a few random questions about wrapper staging:
> >         > 
> >         > 
> >         > How can I make gsiftp:// URIs get sent to
> >         _swiftwrap.wrapperstaging? I
> >         > have wrapper staging enabled, but Swift is trying to
> >         > connect
> >         to the
> >         > server and retrieve the file, rather than just passing
> >         > the
> >         URI to the
> >         > wrapper script.
> >         
> >         With wrapper staging enabled, swift should not try to
> >         *transfer* files,
> >         but mappers might connect to the server to figure out what
> >         files are
> >         there (i.e. to complete calls to Mapper.existing()).
> >         
> >         > 
> >         > What does the property wrapper.staging.local.server do?
> >         
> >         It specifies what url prefix to pass to the wrapper for
> >         local
> >         files
> >         (i.e. accessible directly by swift).
> >         
> >         > 
> >         > 
> >         > From within _swiftwrap.wrapperstaging, how can I know the
> >         current
> >         > working directory of the user? I think I need this
> >         information to
> >         > handle relative path names.
> >         
> >         The wrapper should get a -urlprefix argument containing the
> >         CWD. The
> >         value of that arg is constructed as:
> >         ${wrapper.staging.local.server} +
> >         ${CWD}.
> >         
> >         > 
> >         > 
> >         > I believe in trunk, filenames passed as arguments to
> >         > Swift
> >         apps being
> >         > prepended with "__root__". What does this mean, and
> >         > should
> >         swiftwrap
> >         > strip this out?
> >         
> >         No. There is a mapping from local file names to remote file
> >         names. It
> >         used to be that absolute files are mapped simply by
> >         removing
> >         the leading
> >         slash. So "/dir/file" on the swift side would become
> >         "dir/file" on the
> >         remote side. But then if there was a local "dir/file", it
> >         would also be
> >         "dir/file" on the remote side, causing a conflict.
> >         
> >         At some point in trunk, I changed that (and also the
> >         handling
> >         of "/../")
> >         in response to a bug that, I believe, Ketan submitted. The
> >         change
> >         involved mapping a leading slash to "__root__" and parent
> >         dir
> >         references
> >         to "__parent__". They have no special meaning as far as the
> >         wrapper or
> >         the application is concerned. They get the proper path
> >         through
> >         @filename
> >         and the likes, and swift stages the files in the places
> >         pointed by
> >         @filename.
> >         
> >         Mihael
> >         
> > 
> > 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wilde at mcs.anl.gov  Thu May 16 14:00:54 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 16 May 2013 14:00:54 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <693725412.2475823.1368729761044.JavaMail.root@ci.uchicago.edu>
Message-ID: <1078370649.2767270.1368730854213.JavaMail.root@mcs.anl.gov>

Doesn't "file://" already map to the "local" data provider? If so, can you simply replicate that entry in the provider alias list to create additional aliases to "local" as you suggest below, David?

- Mike

----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Mihael Hategan" <hategan at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Thursday, May 16, 2013 1:42:41 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> 
> 
> I'd like to find a way to define different staging methods on a
> file-by-file basis. In some cases it would make sense to use dd to
> copy a file to the work directory. In other situations it may be
> better to create a symlink pointing to what you need.
> 
> 
> I'm thinking of something along these lines (names will probably get
> changed):
> 
> 
> file data
> <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> // Will stage the data to the work directory
> 
> 
> file data2
> <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> // Will create a symlink
> 
> 
> Then based on the prefix you define, _swiftwrap.wrapperstaging can
> know exactly how to stage the file.
> 
> 
> Is it possible to create data provider aliases, making file-direct://
> and file-indirect:// work exactly like file:// until it gets passed
> to the wrapper?
> 
> 
> Thanks,
> David
> 
> 
> 
> 
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Wednesday, May 8, 2013 12:46:45 PM
> Subject: Re: Wrapper staging
> 
> I'm a bit busy today and this week, so I might not get to fixing
> these
> until the weekend, but I'll try to answer some of your questions.
> 
> On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > Hello,
> > 
> > 
> > Just a few random questions about wrapper staging:
> > 
> > 
> > How can I make gsiftp:// URIs get sent to
> > _swiftwrap.wrapperstaging? I
> > have wrapper staging enabled, but Swift is trying to connect to the
> > server and retrieve the file, rather than just passing the URI to
> > the
> > wrapper script.
> 
> With wrapper staging enabled, swift should not try to *transfer*
> files,
> but mappers might connect to the server to figure out what files are
> there (i.e. to complete calls to Mapper.existing()).
> 
> > 
> > What does the property wrapper.staging.local.server do?
> 
> It specifies what url prefix to pass to the wrapper for local files
> (i.e. accessible directly by swift).
> 
> > 
> > 
> > From within _swiftwrap.wrapperstaging, how can I know the current
> > working directory of the user? I think I need this information to
> > handle relative path names.
> 
> The wrapper should get a -urlprefix argument containing the CWD. The
> value of that arg is constructed as: ${wrapper.staging.local.server}
> +
> ${CWD}.
> 
> > 
> > 
> > I believe in trunk, filenames passed as arguments to Swift apps
> > being
> > prepended with "__root__". What does this mean, and should
> > swiftwrap
> > strip this out?
> 
> No. There is a mapping from local file names to remote file names. It
> used to be that absolute files are mapped simply by removing the
> leading
> slash. So "/dir/file" on the swift side would become "dir/file" on
> the
> remote side. But then if there was a local "dir/file", it would also
> be
> "dir/file" on the remote side, causing a conflict.
> 
> At some point in trunk, I changed that (and also the handling of
> "/../")
> in response to a bug that, I believe, Ketan submitted. The change
> involved mapping a leading slash to "__root__" and parent dir
> references
> to "__parent__". They have no special meaning as far as the wrapper
> or
> the application is concerned. They get the proper path through
> @filename
> and the likes, and swift stages the files in the places pointed by
> @filename.
> 
> Mihael
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From davidk at ci.uchicago.edu  Thu May 16 14:19:02 2013
From: davidk at ci.uchicago.edu (David Kelly)
Date: Thu, 16 May 2013 14:19:02 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <1078370649.2767270.1368730854213.JavaMail.root@mcs.anl.gov>
Message-ID: <2010249710.2488598.1368731942726.JavaMail.root@ci.uchicago.edu>

When I try to use file-direct:// with no modifications I get this: 

Progress: time: Thu, 16 May 2013 19:04:23 +0000 
Execution failed: 
java.lang.RuntimeException: java.lang.RuntimeException: Could not instantiate file resource 
Caused by: 
No 'file-direct' provider or alias found. Available providers: [cobalt, ssh-cl, gsiftp, coaster, webdav, slurm, dcache, ssh, gt2, condor, http, coaster-persistent, pbs, ftp, lsf, gsiftp-old, local, sge]. Aliases: condor <-> condorlocal; cobalt <-> cobaltlocal; gsiftp <-> gridftp; pbs <-> pbslocal; gsiftp-old <-> gridftp-old; local <-> file; 

So I see in that message that local <-> file are aliases. I wasn't sure exactly where those were defined, but I think I found it in modules/provider-local/resources/cog-provider.properties. I added this line: 

alias=file-direct:local 

Then I rebuilt, and it seems to be getting to the wrapper now. Thanks. 

David 
----- Original Message -----

> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan"
> <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:00:54 PM
> Subject: Re: [Swift-devel] Wrapper staging

> Doesn't "file://" already map to the "local" data provider? If so,
> can you simply replicate that entry in the provider alias list to
> create additional aliases to "local" as you suggest below, David?

> - Mike

> ----- Original Message -----
> > From: "David Kelly" <davidk at ci.uchicago.edu>
> > To: "Mihael Hategan" <hategan at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Thursday, May 16, 2013 1:42:41 PM
> > Subject: Re: [Swift-devel] Wrapper staging
> >
> >
> >
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> >
> >
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> >
> >
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> >
> >
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> >
> >
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> >
> >
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to the wrapper?
> >
> >
> > Thanks,
> > David
> >
> >
> >
> >
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> >
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> >
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > >
> > >
> > > Just a few random questions about wrapper staging:
> > >
> > >
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> >
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> >
> > >
> > > What does the property wrapper.staging.local.server do?
> >
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> >
> > >
> > >
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> >
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server}
> > +
> > ${CWD}.
> >
> > >
> > >
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> >
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also
> > be
> > "dir/file" on the remote side, causing a conflict.
> >
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> >
> > Mihael
> >
> >
> >
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130516/530b7cf5/attachment.html>

From wilde at mcs.anl.gov  Thu May 16 14:45:09 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 16 May 2013 14:45:09 -0500 (CDT)
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <2010249710.2488598.1368731942726.JavaMail.root@ci.uchicago.edu>
Message-ID: <802419257.2772040.1368733509453.JavaMail.root@mcs.anl.gov>

Cool.  It might be interesting to see if the "//" is wired-in or optional. If it can be omitted, then you might be able to make the specification more compact and less noisy:

  file data<"direct:/home/davidkelly999/staging/dd-absolute/data.txt">;
  file data<"direct:data.txt">;
  file data<"stage:data.txt">;

- Mike

----- Original Message -----
> From: "David Kelly" <davidk at ci.uchicago.edu>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan" <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:19:02 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> 
> When I try to use file-direct:// with no modifications I get this:
> 
> 
> 
> Progress: time: Thu, 16 May 2013 19:04:23 +0000
> Execution failed:
> java.lang.RuntimeException: java.lang.RuntimeException: Could not
> instantiate file resource
> Caused by:
> No 'file-direct' provider or alias found. Available providers:
> [cobalt, ssh-cl, gsiftp, coaster, webdav, slurm, dcache, ssh, gt2,
> condor, http, coaster-persistent, pbs, ftp, lsf, gsiftp-old, local,
> sge]. Aliases: condor <-> condorlocal; cobalt <-> cobaltlocal;
> gsiftp <-> gridftp; pbs <-> pbslocal; gsiftp-old <-> gridftp-old;
> local <-> file;
> 
> 
> So I see in that message that local <-> file are aliases. I wasn't
> sure exactly where those were defined, but I think I found it in
> modules/provider-local/resources/cog-provider.properties. I added
> this line:
> 
> 
> alias=file-direct:local
> 
> 
> Then I rebuilt, and it seems to be getting to the wrapper now.
> Thanks.
> 
> 
> David
> ----- Original Message -----
> 
> 
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Mihael Hategan"
> <hategan at mcs.anl.gov>
> Sent: Thursday, May 16, 2013 2:00:54 PM
> Subject: Re: [Swift-devel] Wrapper staging
> 
> Doesn't "file://" already map to the "local" data provider? If so,
> can you simply replicate that entry in the provider alias list to
> create additional aliases to "local" as you suggest below, David?
> 
> - Mike
> 
> ----- Original Message -----
> > From: "David Kelly" <davidk at ci.uchicago.edu>
> > To: "Mihael Hategan" <hategan at mcs.anl.gov>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Thursday, May 16, 2013 1:42:41 PM
> > Subject: Re: [Swift-devel] Wrapper staging
> > 
> > 
> > 
> > I'd like to find a way to define different staging methods on a
> > file-by-file basis. In some cases it would make sense to use dd to
> > copy a file to the work directory. In other situations it may be
> > better to create a symlink pointing to what you need.
> > 
> > 
> > I'm thinking of something along these lines (names will probably
> > get
> > changed):
> > 
> > 
> > file data
> > <"file-direct:///home/davidkelly999/staging/dd-absolute/data.txt">;
> > // Will stage the data to the work directory
> > 
> > 
> > file data2
> > <"file-indirect:///home/davidkelly999/staging/dd-absolute/data2.txt">;
> > // Will create a symlink
> > 
> > 
> > Then based on the prefix you define, _swiftwrap.wrapperstaging can
> > know exactly how to stage the file.
> > 
> > 
> > Is it possible to create data provider aliases, making
> > file-direct://
> > and file-indirect:// work exactly like file:// until it gets passed
> > to the wrapper?
> > 
> > 
> > Thanks,
> > David
> > 
> > 
> > 
> > 
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Wednesday, May 8, 2013 12:46:45 PM
> > Subject: Re: Wrapper staging
> > 
> > I'm a bit busy today and this week, so I might not get to fixing
> > these
> > until the weekend, but I'll try to answer some of your questions.
> > 
> > On Wed, 2013-05-08 at 10:53 -0500, David Kelly wrote:
> > > Hello,
> > > 
> > > 
> > > Just a few random questions about wrapper staging:
> > > 
> > > 
> > > How can I make gsiftp:// URIs get sent to
> > > _swiftwrap.wrapperstaging? I
> > > have wrapper staging enabled, but Swift is trying to connect to
> > > the
> > > server and retrieve the file, rather than just passing the URI to
> > > the
> > > wrapper script.
> > 
> > With wrapper staging enabled, swift should not try to *transfer*
> > files,
> > but mappers might connect to the server to figure out what files
> > are
> > there (i.e. to complete calls to Mapper.existing()).
> > 
> > > 
> > > What does the property wrapper.staging.local.server do?
> > 
> > It specifies what url prefix to pass to the wrapper for local files
> > (i.e. accessible directly by swift).
> > 
> > > 
> > > 
> > > From within _swiftwrap.wrapperstaging, how can I know the current
> > > working directory of the user? I think I need this information to
> > > handle relative path names.
> > 
> > The wrapper should get a -urlprefix argument containing the CWD.
> > The
> > value of that arg is constructed as:
> > ${wrapper.staging.local.server}
> > +
> > ${CWD}.
> > 
> > > 
> > > 
> > > I believe in trunk, filenames passed as arguments to Swift apps
> > > being
> > > prepended with "__root__". What does this mean, and should
> > > swiftwrap
> > > strip this out?
> > 
> > No. There is a mapping from local file names to remote file names.
> > It
> > used to be that absolute files are mapped simply by removing the
> > leading
> > slash. So "/dir/file" on the swift side would become "dir/file" on
> > the
> > remote side. But then if there was a local "dir/file", it would
> > also
> > be
> > "dir/file" on the remote side, causing a conflict.
> > 
> > At some point in trunk, I changed that (and also the handling of
> > "/../")
> > in response to a bug that, I believe, Ketan submitted. The change
> > involved mapping a leading slash to "__root__" and parent dir
> > references
> > to "__parent__". They have no special meaning as far as the wrapper
> > or
> > the application is concerned. They get the proper path through
> > @filename
> > and the likes, and swift stages the files in the places pointed by
> > @filename.
> > 
> > Mihael
> > 
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > 
> 
> 


From wilde at mcs.anl.gov  Thu May 16 18:08:39 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 16 May 2013 18:08:39 -0500 (CDT)
Subject: [Swift-devel] dynamic site selection?
In-Reply-To: <1793569778.2858505.1368745317351.JavaMail.root@mcs.anl.gov>
Message-ID: <338959050.2859616.1368745719199.JavaMail.root@mcs.anl.gov>


Do we have any mechanism that can be used to for an app() invocation to select a specific site?

The immediate application for this is to use Swift itself to drive the remote Swift site testing. 

One way to do this is with something like dynamic profiles, which forces an app() invocation to select a specific site name (assuming the app is cataloged on that site). Eg, a "runtime" settings directive that works like the "profile" directive and passes a site to the scheduler:

app (file log) runTest (file test_package, string site)
{
  runtime "site" = site;
  runTestSuite @log @test_package;
}

string sites["midway","fusion","uc3","beagle","blues"];
...

foreach site in sites {
  log[site] = runTest(testpack[site],site);
}

A perhaps simpler interim step would be the ability to set the command name in the app command template dynamically, thus selecting one of a set of apps cataloged in the tc.data file:

app (file log) runTest (file test_package, string site)
{
  @strcat("runTestSuite_",site) @log @test_package;
}

This avoids having to do the same thing with a set of N app() definitions, one per site, and a routing function with a switch statement. Which is what we'll have to do in the interim.

Are any of these solutions easy to do?

Thanks,

- Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Thu May 16 18:32:43 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Thu, 16 May 2013 16:32:43 -0700
Subject: [Swift-devel] Wrapper staging
In-Reply-To: <2010249710.2488598.1368731942726.JavaMail.root@ci.uchicago.edu>
References: <2010249710.2488598.1368731942726.JavaMail.root@ci.uchicago.edu>
Message-ID: <1368747163.26509.1.camel@echo>

On Thu, 2013-05-16 at 14:19 -0500, David Kelly wrote:
> When I try to use file-direct:// with no modifications I get this: 
> 
> Progress: time: Thu, 16 May 2013 19:04:23 +0000 
> Execution failed: 
> java.lang.RuntimeException: java.lang.RuntimeException: Could not instantiate file resource 
> Caused by: 
> No 'file-direct' provider or alias found. Available providers: [cobalt, ssh-cl, gsiftp, coaster, webdav, slurm, dcache, ssh, gt2, condor, http, coaster-persistent, pbs, ftp, lsf, gsiftp-old, local, sge]. Aliases: condor <-> condorlocal; cobalt <-> cobaltlocal; gsiftp <-> gridftp; pbs <-> pbslocal; gsiftp-old <-> gridftp-old; local <-> file; 

That's what I meant by the mapper doing stuff.

> 
> So I see in that message that local <-> file are aliases. I wasn't sure exactly where those were defined, but I think I found it in modules/provider-local/resources/cog-provider.properties. I added this line: 
> 
> alias=file-direct:local 

That should work. However, I think this needs to be fixed in swift
eventually.

Mihael




From wilde at mcs.anl.gov  Wed May 22 14:16:07 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 22 May 2013 14:16:07 -0500 (CDT)
Subject: [Swift-devel] Attributes vs Options in coaster-c-client
In-Reply-To: <1241590796.1026975.1369248841295.JavaMail.root@mcs.anl.gov>
Message-ID: <448710485.1043440.1369250167323.JavaMail.root@mcs.anl.gov>


In the coaster-c-client code, run-coaster-job -help has these two items:

        --attr|-a <name>=<value>
                      A job attribute (such as 'maxwalltime').
                      Can be used more than once.

        --option|-o <name>=<value>
                      An option to pass to the coaster service
                      (such as 'slots=10'). Can be used more than
                      once.
The two mentioned in the "help" message are specified in the same manner in sites.xml:

    <profile namespace="globus" key="maxWalltime">00:05:00</profile>
    <profile namespace="globus" key="slots">4</profile>

How are attrs and options different and how do we determine which sites attribute is set in which way?

- Mike

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From hategan at mcs.anl.gov  Wed May 22 14:36:26 2013
From: hategan at mcs.anl.gov (Mihael Hategan)
Date: Wed, 22 May 2013 12:36:26 -0700
Subject: [Swift-devel] Attributes vs Options in coaster-c-client
In-Reply-To: <448710485.1043440.1369250167323.JavaMail.root@mcs.anl.gov>
References: <448710485.1043440.1369250167323.JavaMail.root@mcs.anl.gov>
Message-ID: <1369251386.20038.7.camel@echo>

An option is something that is used to configure the entire service,
whereas an attribute is job specific.

In the case below, slots is something that applies to all the jobs after
the service is started, while maxwalltime can be different for different
jobs.

Internally, in the java code, all of these end up as attributes of the
task object, but the code extracts the ones it knows are service options
before submission of the first job and uses them to configure the
service.

Options are passed using the Settings class, so you can look in
Settings.cpp for the keys. They correspond to the info in the Java
version of Settings, and are described somewhere in the swift guide
(Section 18.4).

Mihael

On Wed, 2013-05-22 at 14:16 -0500, Michael Wilde wrote:
> In the coaster-c-client code, run-coaster-job -help has these two items:
> 
>         --attr|-a <name>=<value>
>                       A job attribute (such as 'maxwalltime').
>                       Can be used more than once.
> 
>         --option|-o <name>=<value>
>                       An option to pass to the coaster service
>                       (such as 'slots=10'). Can be used more than
>                       once.
> The two mentioned in the "help" message are specified in the same manner in sites.xml:
> 
>     <profile namespace="globus" key="maxWalltime">00:05:00</profile>
>     <profile namespace="globus" key="slots">4</profile>
> 
> How are attrs and options different and how do we determine which sites attribute is set in which way?
> 
> - Mike
> 




From ketancmaheshwari at gmail.com  Thu May 23 23:08:28 2013
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 23 May 2013 23:08:28 -0500
Subject: [Swift-devel] ssh-cl:pbs from communicado to fusion
In-Reply-To: <CAMUuvioZC_P7kQfKCTi+BKgC8m-iL+9GKe1Tp2kVZfYnDj9O=w@mail.gmail.com>
References: <CAMUuvioZC_P7kQfKCTi+BKgC8m-iL+9GKe1Tp2kVZfYnDj9O=w@mail.gmail.com>
Message-ID: <CAMUuvipP9SS0yTAR9Px8kgrsVYZBvxaW5u4GEYXRgQh9LSUrwA@mail.gmail.com>

Please Disregard.

This has worked after commenting the following lines in .bashrc on fusion:

#export X509_CERT_DIR=/home/ketan/TRUSTEDCA
#export X509_CADIR=/home/ketan/TRUSTEDCA



On Thu, May 23, 2013 at 11:03 PM, Ketan Maheshwari <
ketancmaheshwari at gmail.com> wrote:

> Hi,
>
> Trying ssh-cl from communicado into fusion. Fails with the following on
> coaster bootstrap log on fusion side:
>
> using plain mode
> BS: http://communicado.ci.uchicago.edu:50001
> Failed to download bootstrap jar from
> http://communicado.ci.uchicago.edu:50001
>
> sites pool entry is as follows:
>
> <pool handle="fusion">
>   <execution provider="coaster" jobmanager="ssh-cl:pbs" url="
> fusion.lcrc.anl.gov" />
>   <profile namespace="globus" key="maxtime">800</profile>
>   <profile namespace="globus" key="maxWalltime">00:10:00</profile>
>   <profile namespace="globus" key="jobsPerNode">1</profile>
>   <profile namespace="globus" key="count">1</profile>
>   <profile namespace="globus" key="slots">1</profile>
>   <profile namespace="globus" key="ppn">8</profile>
>   <profile namespace="karajan" key="jobThrottle">.15</profile>
>   <profile namespace="karajan" key="initialScore">10000</profile>
>   <filesystem provider="local"/>
>   <workdirectory>/tmp</workdirectory>
> </pool>
>
> cf is as follows:
> wrapperlog.always.transfer=true
> sitedir.keep=true
> execution.retries=0
> lazy.errors=false
> status.mode=provider
> use.provider.staging=true
> provider.staging.pin.swiftfiles=false
>
>
> The same setup works with another lcrc machine blues and a remote machine
> with ssh-cl:local.
>
> std.err attached. Any clues?
>
> --
> Ketan
>
>


-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130523/c82849e7/attachment.html>

From ketancmaheshwari at gmail.com  Thu May 23 23:03:38 2013
From: ketancmaheshwari at gmail.com (Ketan Maheshwari)
Date: Thu, 23 May 2013 23:03:38 -0500
Subject: [Swift-devel] ssh-cl:pbs from communicado to fusion
Message-ID: <CAMUuvioZC_P7kQfKCTi+BKgC8m-iL+9GKe1Tp2kVZfYnDj9O=w@mail.gmail.com>

Hi,

Trying ssh-cl from communicado into fusion. Fails with the following on
coaster bootstrap log on fusion side:

using plain mode
BS: http://communicado.ci.uchicago.edu:50001
Failed to download bootstrap jar from
http://communicado.ci.uchicago.edu:50001

sites pool entry is as follows:

<pool handle="fusion">
  <execution provider="coaster" jobmanager="ssh-cl:pbs" url="
fusion.lcrc.anl.gov" />
  <profile namespace="globus" key="maxtime">800</profile>
  <profile namespace="globus" key="maxWalltime">00:10:00</profile>
  <profile namespace="globus" key="jobsPerNode">1</profile>
  <profile namespace="globus" key="count">1</profile>
  <profile namespace="globus" key="slots">1</profile>
  <profile namespace="globus" key="ppn">8</profile>
  <profile namespace="karajan" key="jobThrottle">.15</profile>
  <profile namespace="karajan" key="initialScore">10000</profile>
  <filesystem provider="local"/>
  <workdirectory>/tmp</workdirectory>
</pool>

cf is as follows:
wrapperlog.always.transfer=true
sitedir.keep=true
execution.retries=0
lazy.errors=false
status.mode=provider
use.provider.staging=true
provider.staging.pin.swiftfiles=false


The same setup works with another lcrc machine blues and a remote machine
with ssh-cl:local.

std.err attached. Any clues?

-- 
Ketan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130523/ac3ab470/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: std.err
Type: application/octet-stream
Size: 5189 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130523/ac3ab470/attachment.obj>

From iraicu at cs.iit.edu  Fri May 24 16:38:46 2013
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Fri, 24 May 2013 16:38:46 -0500
Subject: [Swift-devel] Call for Participation: ACM ScienceCloud 2013 @ HPDC
	2013
Message-ID: <519FDDE6.2080204@cs.iit.edu>


    Call for Participation

**


    4th Workshop on Scientific Cloud Computing (ScienceCloud) 2013

*Co-located with ACM HPDC 2013 <http://www.hpdc.org/2013/>* *
New York City, NY, USA -- June 17th, 2013
**


      http://datasys.cs.iit.edu/events/ScienceCloud2013/


      Workshop Program

*

  * *Date: June 17th, 2013
  * * *Time: 9AM - 5PM*

The workshop features a keynote talk by Dr. Ian T. Foster 
(UChicago/ANL), an invited talk by Radu Sion (Stony Brook), and 7 papers 
and oral presentations. For more information on the keynote and invited 
speakers, click here 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/program.html#Invited_speakers_biographies:>. 


Time 	Description 	Presenter 	Institution
9:00AM 	Opening Remarks 	
	
9:05AM 	Keynote -- Science as a Service: How On-Demand Computing Can 
Accelerate Discovery 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/keynote.html> 
(Abstract 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/keynote.pdf>, 
Slides) 	Ian T. Foster 	ANL
UChicago
10:00AM 	Break 	
	

	Session 1: Applications and Services 	
	
10:30AM 	High Performance Risk Aggregation: Addressing the Data 
Processing Challenge the Hadoop MapReduce Way (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p01.pdf>, Slides) 	A. 
Rau-Chaplin
B. Varghese
Z. Yao 	Dalhousie Univ.
11:00AM 	Performance Evaluation of a MongoDB and Hadoop Platform for 
Scientific Data Analysis (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p02.pdf>, Slides) 	E. 
Dede
M. Govindaraju
D. Gunter
R. Canon
L. Ramakrishnan 	SUNY
LBL
11:30AM 	VIDAS: Object-based Virtualized Data Sharing for High 
Performance Storage I/O (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p03.pdf>, Slides) 
Pablo Llopis
Javier Garcia Blas
Florin Isaila
Jesus Carretero 	Univ. Carlos III de Madrid
12:00PM 	Lunch 	
	

	Session 2: Cloud Services 	
	
1:30PM 	Invited talk -- To Cloud My Big Data or Not To? Musings at the 
Intersection of Big Data, Intense Computing andClouds 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/invited.html> 
(Abstract 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/invited.html>, 
Slides) 	Radu Sion 	Stony Brook
2:30PM 	StorkCloud: Data Transfer Scheduling and Optimization as 
aService (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p04.pdf>, Slides) 
Tevfik Kosar
Engin Arslan
Brandon Ross
Bing Zhang 	University at Buffalo
3:00PM 	Break 	
	

	Session 3: Technology and Infrastructure 	
	
3:30PM 	Rebalancing in a Multi-Cloud Environment (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p05.pdf>, Slides) 
Dmitry Duplyakin
Ali Alzabarah
Paul Marshall
Kate Keahey
Henry Tufo 	Univ. of Colorado, Boulder
ANL
UChicago
4:00PM 	Dimensioning the Virtual Cluster for Parallel Scientific 
Workflows in Clouds (Paper 
<http://datasys.cs.iit.edu/events/ScienceCloud2013/p06.pdf>, Slides) 
Daniel de Oliveira
Vitor Viana
Eduardo Ogasawara
Kary Oca?a
Marta Mattoso 	COPPE/UFRJ
CEFET/RJ
4:30PM 	HTC Scientific Computing in a Distributed Cloud Environment 
(Paper <http://datasys.cs.iit.edu/events/ScienceCloud2013/p07.pdf>, 
Slides) 	R.J. Sobie
A. Agarwal
I. Gable
C. Leavett-Brown
M. Paterson
R. Taylor
A. Charbonneau
R. Impey
W. Podiama 	Univ. of Victoria
NRC Canada
5:00PM 	Closing Remarks 	
	

*****


        Invited speakers biographies:

* * ** * *

Keynote

*

Ian FosterDr. Ian t. Foster, Professor at Unviersity of Chicago, 
Director at Computation Institute, & Fellow at Argonne National 
Labotaroty <http://www.mcs.anl.gov/about/people_detail.php?id=285>

Dr. Ian Foster <http://www.mcs.anl.gov/about/people_detail.php?id=285> 
is Director of the Computation Institute, a joint institute of the 
University of Chicago and Argonne National Laboratory. He is also an 
Argonne Senior Scientist and Distinguished Fellow, and the Arthur Holly 
Compton Distinguished Service Professor of Computer Science at 
University of Chicago. He is also involved with both the Open Grid Forum 
and with the Globus Alliance as an open source strategist. In 2006, he 
was appointed director of the Computation Institute, a joint project 
between the University of Chicago, and Argonne. An earlier project, 
Strand, received the British Computer Society Award for technical 
innovation. His research resulted in the development of techniques, 
tools and algorithms for high-performance distributed computing and 
parallel computing. As a result he is denoted as "the father of the 
Grid". Foster led research and development of software for the I-WAY 
wide-area distributed computing experiment, which connected 
supercomputers, databases and other high-end resources at 17 sites 
across North America in 1995. His own labs, the Distributed Systems 
Laboratory is the nexus of the multi-institute Globus Project, a 
research and development effort that encourages collaborative computing 
by providing advances necessary for engineering, business and other 
fields. Furthermore the Computation Institute addresses many of the most 
challenging computational and communications problems facing Grid 
implementations today. In 2004, he founded Univa Corporation, which was 
merged with United Devices in 2007 and operate under the name Univa UD. 
Foster's honors include the Lovelace Medal of the British Computer 
Society, the Gordon Bell Prize for high-performance computing (2001), as 
well as others. He was elected Fellow of the American Association for 
the Advancement of Science in 2003. Dr. Foster also serves as PI or 
Co-PI on projects connected to the DOE global change program, the 
National Computational Science Alliance, the NASA Information Power Grid 
project, the NSF Grid Physics Network, GRIDS Center, and International 
Virtual Data Grid Laboratory projects, and other DOE and NSF programs. 
His research is supported by DOE, NSF, NASA, Microsoft, and IBM.

*

Invited Speaker

*

Dr. Radu Sion, Associate Professor at Stony Brook University & CEO of 
Private Machines Inc. <http://digitalpiglet.org>

Radu Sion <http://digitalpiglet.org> is an Associate Professor of 
Computer Science at Stony Brook University (on leave) and currently the 
CEO of Private Machines Inc. He remembers when gophers were digging 
through the Internets and bits were running at slower paces of 512 per 
second. He is also interested in efficient computing with a touch of 
cyber-security paranoia, raising rabbits on space ships and sailing 
catamarans of the Hobie variety. **




-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer JoCCASA
Chair:  IEEE/ACM MTAGS, ACM ScienceCloud, IEEE/ACM DataCloud
=================================================================
Cel:      1-847-722-0876
Office:   1-312-567-5704
Email:    iraicu at cs.iit.edu
Web:      http://www.cs.iit.edu/~iraicu/
Web:      http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google:   http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130524/951cbaee/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Ian_Foster.jpg
Type: image/jpeg
Size: 12848 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130524/951cbaee/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sion.jpg
Type: image/jpeg
Size: 33686 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130524/951cbaee/attachment-0001.jpg>

From tim.g.armstrong at gmail.com  Fri May 24 17:01:33 2013
From: tim.g.armstrong at gmail.com (Tim Armstrong)
Date: Fri, 24 May 2013 17:01:33 -0500
Subject: [Swift-devel] Domain name
Message-ID: <CAC0jiV4+P-ieh=a5v5gX+JNTCCDu5VtQeJmquXAiA6LPjFJQ3A@mail.gmail.com>

Following up on the earlier discussion, swift-lang.org isn't taken.

- Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130524/bdbe2953/attachment.html>

From itf at mcs.anl.gov  Fri May 24 17:06:06 2013
From: itf at mcs.anl.gov (Ian Foster)
Date: Fri, 24 May 2013 17:06:06 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <CAC0jiV4+P-ieh=a5v5gX+JNTCCDu5VtQeJmquXAiA6LPjFJQ3A@mail.gmail.com>
References: <CAC0jiV4+P-ieh=a5v5gX+JNTCCDu5VtQeJmquXAiA6LPjFJQ3A@mail.gmail.com>
Message-ID: <022DC1F0-087E-41BD-8945-02502BE8B155@anl.gov>

Nice idea 

On May 24, 2013, at 5:01 PM, Tim Armstrong <tim.g.armstrong at gmail.com> wrote:

> Following up on the earlier discussion, swift-lang.org isn't taken.
> 
> - Tim
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20130524/56780e02/attachment.html>

From wilde at mcs.anl.gov  Fri May 24 19:25:27 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 24 May 2013 19:25:27 -0500 (CDT)
Subject: [Swift-devel] Domain name
In-Reply-To: <CAC0jiV4+P-ieh=a5v5gX+JNTCCDu5VtQeJmquXAiA6LPjFJQ3A@mail.gmail.com>
Message-ID: <1192744849.2045978.1369441527231.JavaMail.root@mcs.anl.gov>

I got swiftlang.org and swift-lang.org

We can initially forward them to the www.ci.uchicago.edu/swift site, then later get the site to appear as if its hosted at the new domain name.

- Mike

----- Original Message -----
> From: "Tim Armstrong" <tim.g.armstrong at gmail.com>
> To: "swift-devel" <swift-devel at ci.uchicago.edu>
> Sent: Friday, May 24, 2013 5:01:33 PM
> Subject: [Swift-devel] Domain name
> 
> 
> 
> 
> Following up on the earlier discussion, swift-lang.org isn't taken.
> 
> 
> - Tim
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 


From wilde at mcs.anl.gov  Thu May 30 19:50:01 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Thu, 30 May 2013 19:50:01 -0500 (CDT)
Subject: [Swift-devel] back-level 0.94 release on Midway?
Message-ID: <1669051623.1368927.1369961401368.JavaMail.root@mcs.anl.gov>


David, on beagle we have:

$ swift -version                                                                                                               
Swift 0.94 swift-r6492 cog-r3658

But on Midway we have:

$ swift -version
Swift 0.94 swift-r6414 (swift modified locally) cog-r3648

Can you bring midway up to the final release level for 0.94, or is there a reason its not?

Thanks,

- Mike




From wilde at mcs.anl.gov  Fri May 31 11:29:36 2013
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 31 May 2013 11:29:36 -0500 (CDT)
Subject: [Swift-devel] Coaster provider staging timeout problems lingering
In-Reply-To: <1716155600.154516.1363096261076.JavaMail.root@mcs.anl.gov>
Message-ID: <913312318.1540830.1370017776370.JavaMail.root@mcs.anl.gov>

I have heard from David a while back and Yadu this week that coaster provider staging timeouts remain a problem, albeit a far less frequent one.

Yadu seems to have a test (which he is polishing) that can reproduce the problem readily.

Yadu, do the coaster provider staging timeouts that you are seeing on the ex-search app match what I encountered back in March on OSG via UC3 (below)?

Please either file a ticket asap and assign to Mihael or locate an existing ticket for this bug and add more logs and incidents to it.

David, can you tell us what your impression of remaining provider staging failure scenarios are, ideally via an existing ticket if there's one open for this?

Thanks,

- Mike

----- Forwarded Message -----
From: "Michael Wilde" <wilde at mcs.anl.gov>
To: "Mihael Hategan" <hategan at mcs.anl.gov>
Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
Sent: Tuesday, March 12, 2013 8:51:01 AM
Subject: [Swift-devel] Coaster run to UC3 dies with channel timeout

This demo (for OSG all-hands) was running fairly reliably, 100's to a few thousand 30-second tasks to UC3 with flocking to OSG and other pools.

But just got a failure, so it looks like sporadic problems remain.

Running Swift 0.94 latest rev.

Log is on midway in:

/home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
-rw-rw-r-- 1 wilde wilde 11632001 Mar 12 08:42 saved/modis-20130312-1335-p30ylps9.log

I'll file a ticket once we get a sense of the frequency.

- Mike



 Progress:  time: Tue, 12 Mar 2013 13:42:28 +0000  Selecting site:461  Stage in:10  Submitted:782  Active:204  Stage out:4  Finished successfully:1539
Progress:  time: Tue, 12 Mar 2013 13:42:29 +0000  Selecting site:453  Stage in:6  Submitted:779  Active:215  Finished successfully:1547
Progress:  time: Tue, 12 Mar 2013 13:42:30 +0000  Selecting site:439  Stage in:16  Submitting:1  Submitted:776  Active:204  Stage out:2  Finished successfully:1562
Execution failed:
	Exception in perl:
    Arguments: [getlanduse.pl, input/h06v33.rgb]
    Host: uc3
    Directory: modis-20130312-1335-p30ylps9/jobs/7/perl-7s7qvh6l

Caused by:
	Task failed: null
org.globus.cog.karajan.workflow.service.TimeoutException: Channel timed out. lastTime=130312-084030.762, now=130312-084231.763, channel=TCP-0312-3508510-000259-000000
	at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:131)
	at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:122)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)

	getLandUse, modis.swift, line 24
swift$ pwd
/home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
swift$ ls
cf		input/	  modis-20130312-1335-p30ylps9.0.rlog  modis-20130312-1335-p30ylps9.log  saved/     tc
getlanduse.pl*	landuse/  modis-20130312-1335-p30ylps9.d/      run*				 swift.log  uc3.xml
swift$ e ../save
swift$ save
swift$ ls saved
modis-20130312-1326-n9rofj6e.d/   modis-20130312-1329-f2a2eic4.log     modis-20130312-1335-p30ylps9.log
modis-20130312-1326-n9rofj6e.log  modis-20130312-1335-p30ylps9.0.rlog  swift.log
modis-20130312-1329-f2a2eic4.d/   modis-20130312-1335-p30ylps9.d/
swift$ ls saved/modis-20130312-1335-p30ylps9.log
saved/modis-20130312-1335-p30ylps9.log
swift$ pwd; ls -l saved/modis-20130312-1335-p30ylps9.log
/home/wilde/osgdemo/modis/svn/swiftdemo/test.uc3
-rw-rw-r-- 1 wilde wilde 11632001 Mar 12 08:42 saved/modis-20130312-1335-p30ylps9.log
swift$ 


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

_______________________________________________
Swift-devel mailing list
Swift-devel at ci.uchicago.edu
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel