From jmargolpeople at gmail.com  Tue Jul  3 16:48:16 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Tue, 3 Jul 2012 16:48:16 -0500
Subject: [Swift-user] How does swift mapping to files that do not yet exist
	work?
Message-ID: <CAJgtAsosxNpBhr8wdFnhEKjxVbNzhJbr-1DpHT=JEmY+eY=x6Q@mail.gmail.com>

Hello swift user base,

I have been having trouble with a bit of swift code for a while now, and
despite my hesitancy to ask other people to debug my code, I believe my
issue may be more of a lack of understanding of swift than a simple syntax
error, so I have decided to send out this email. Specifically, I have a
single_file_mapper mapping to a file which does not yet exist when my code
is run, in hopes that the file being mapped to will be written to by a an
app call. However, when I try running my code, I get the error back from
swift

Execution failed:
    File not found: *filename *(in this case new_point1_1.mat)
*
*This paradigm of referencing files that do not yet exist has worked before
in swift, so I do not know what the problem is here. Hopefully it is not a
simple syntax bug. Anyway, thank you for your help,

Jonathan

----- My Code: ----------

#cce_ua.swift
type file;

int num_complexes = @toint(@arg("num_complexes"));
int num_simplexes_per_complex = @toint(@arg("num_simplexes_per_complex"));

app (file output, file log) cce_ua (file input1, file input2, int number,
file input3)
{
   cce_ua @input1 @input2 @output number @input3 stdout=@filename(log)
stderr=@filename(log);
   #located at _RUNDIR_/swat/cce_ua.sh
}

file inSCE<single_file_mapper;file="SCE_Par.mat">;
file inrunswat<single_file_mapper;file="RunSwat.m">;

foreach i in [1:num_complexes]
{
    foreach j in [1:num_simplexes_per_complex]
    {
        # These files already exist in the local directory
        file input<single_file_mapper; file=@strcat("simplex", i, "_", j,
".mat")>;
        # These files do not exist in the local directory at runtime
        file output<single_file_mapper; file=@strcat("new_point", i, "_",
j, ".mat")>;
        file log<single_file_mapper; file=@strcat("cce_ua", i, "_", j,
"_log.txt")>;

        (output, log) = cce_ua(inSCE, input, i*num_simplexes_per_complex +
j, inrunswat);
    }
}


---- The command line call to swift (broken up over multiple lines) ---
swift -tc.file swift_files/executable_locations -sites.file
swift_files/sites.xml cce_ua.swift -num_complexes=2
-num_simplexes_per_complex=1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120703/be1ae8d8/attachment.html>

From wilde at mcs.anl.gov  Tue Jul  3 17:11:45 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 3 Jul 2012 17:11:45 -0500 (CDT)
Subject: [Swift-user] How does swift mapping to files that do not yet
 exist	work?
In-Reply-To: <CAJgtAsosxNpBhr8wdFnhEKjxVbNzhJbr-1DpHT=JEmY+eY=x6Q@mail.gmail.com>
Message-ID: <380990982.6730.1341353505963.JavaMail.root@zimbra.anl.gov>

Jonathan, from a cursory read of your script, I suspect that the Swift runtime is complaining that the app cce_ua did not produce the file new_point1_1.mat, which your output file mapping specified.

Given your output mapping, Swift is expecting new_point1_1.mat to exist in the temporary job directory in which your app was launched. I suspect your app (matlab script) did not create a file of this exact name.

You can specify the property "sitedir.keep=true" in your config file (e.g., -config cf on the command line) to tell Swift to retain these job directories on failure. They should be below the "workdirectory" that you specified in your sites.xml file.
(I think...)

- Mike


----- Original Message -----
> From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> To: swift-user at ci.uchicago.edu
> Sent: Tuesday, July 3, 2012 4:48:16 PM
> Subject: [Swift-user] How does swift mapping to files that do not yet exist work?
> Hello swift user base,
> 
> I have been having trouble with a bit of swift code for a while now,
> and despite my hesitancy to ask other people to debug my code, I
> believe my issue may be more of a lack of understanding of swift than
> a simple syntax error, so I have decided to send out this email.
> Specifically, I have a single_file_mapper mapping to a file which does
> not yet exist when my code is run, in hopes that the file being mapped
> to will be written to by a an app call. However, when I try running my
> code, I get the error back from swift
> 
> Execution failed:
> File not found: filename (in this case new_point1_1.mat)
> 
> This paradigm of referencing files that do not yet exist has worked
> before in swift, so I do not know what the problem is here. Hopefully
> it is not a simple syntax bug. Anyway, thank you for your help,
> 
> Jonathan
> 
> ----- My Code: ----------
> 
> #cce_ua.swift
> type file;
> 
> int num_complexes = @toint(@arg("num_complexes"));
> int num_simplexes_per_complex =
> @toint(@arg("num_simplexes_per_complex"));
> 
> app (file output, file log) cce_ua (file input1, file input2, int
> number, file input3)
> {
> cce_ua @input1 @input2 @output number @input3 stdout=@filename(log)
> stderr=@filename(log);
> #located at _RUNDIR_/swat/cce_ua.sh
> }
> 
> file inSCE<single_file_mapper;file="SCE_Par.mat">;
> file inrunswat<single_file_mapper;file="RunSwat.m">;
> 
> foreach i in [1:num_complexes]
> {
> foreach j in [1:num_simplexes_per_complex]
> {
> # These files already exist in the local directory
> file input<single_file_mapper; file=@strcat("simplex", i, "_", j,
> ".mat")>;
> # These files do not exist in the local directory at runtime
> file output<single_file_mapper; file=@strcat("new_point", i, "_", j,
> ".mat")>;
> file log<single_file_mapper; file=@strcat("cce_ua", i, "_", j,
> "_log.txt")>;
> 
> (output, log) = cce_ua(inSCE, input, i*num_simplexes_per_complex + j,
> inrunswat);
> }
> }
> 
> 
> ---- The command line call to swift (broken up over multiple lines)
> ---
> swift -tc.file swift_files/executable_locations -sites.file
> swift_files/sites.xml cce_ua.swift -num_complexes=2
> -num_simplexes_per_complex=1
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jmargolpeople at gmail.com  Tue Jul  3 17:24:19 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Tue, 3 Jul 2012 17:24:19 -0500
Subject: [Swift-user] How does swift mapping to files that do not yet
 exist work?
In-Reply-To: <380990982.6730.1341353505963.JavaMail.root@zimbra.anl.gov>
References: <CAJgtAsosxNpBhr8wdFnhEKjxVbNzhJbr-1DpHT=JEmY+eY=x6Q@mail.gmail.com>
	<380990982.6730.1341353505963.JavaMail.root@zimbra.anl.gov>
Message-ID: <CAJgtAsouCAmQ7DrGAsSmdH3+7ELgNqSRdYE4nojy4GHRxX6tdQ@mail.gmail.com>

Sorry, I should have specified this earlier, but I do not believe my cce_ua
function was ever called. The first line of my cce_ua function is a
statement that prints to stdout. The second and third lines open a file
(via absolute path) and write to it. I have not observed either of those
being executed. Furthermore, the following (matlab) code process the output
argument:

output_point_location = argv(){3};

new_point = ....

save(output_point_location, 'new_point');

There is no other code that modifies the output_point_location variable. So
shouldn't this work, given how swift passes files as command line arguments?

I think something else is the problem, but of course any solution would be
great. Thanks!

Jonathan

On Tue, Jul 3, 2012 at 5:11 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:

> Jonathan, from a cursory read of your script, I suspect that the Swift
> runtime is complaining that the app cce_ua did not produce the file
> new_point1_1.mat, which your output file mapping specified.
>
> Given your output mapping, Swift is expecting new_point1_1.mat to exist in
> the temporary job directory in which your app was launched. I suspect your
> app (matlab script) did not create a file of this exact name.
>
> You can specify the property "sitedir.keep=true" in your config file
> (e.g., -config cf on the command line) to tell Swift to retain these job
> directories on failure. They should be below the "workdirectory" that you
> specified in your sites.xml file.
> (I think...)
>
> - Mike
>
>
> ----- Original Message -----
> > From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> > To: swift-user at ci.uchicago.edu
> > Sent: Tuesday, July 3, 2012 4:48:16 PM
> > Subject: [Swift-user] How does swift mapping to files that do not yet
> exist work?
> > Hello swift user base,
> >
> > I have been having trouble with a bit of swift code for a while now,
> > and despite my hesitancy to ask other people to debug my code, I
> > believe my issue may be more of a lack of understanding of swift than
> > a simple syntax error, so I have decided to send out this email.
> > Specifically, I have a single_file_mapper mapping to a file which does
> > not yet exist when my code is run, in hopes that the file being mapped
> > to will be written to by a an app call. However, when I try running my
> > code, I get the error back from swift
> >
> > Execution failed:
> > File not found: filename (in this case new_point1_1.mat)
> >
> > This paradigm of referencing files that do not yet exist has worked
> > before in swift, so I do not know what the problem is here. Hopefully
> > it is not a simple syntax bug. Anyway, thank you for your help,
> >
> > Jonathan
> >
> > ----- My Code: ----------
> >
> > #cce_ua.swift
> > type file;
> >
> > int num_complexes = @toint(@arg("num_complexes"));
> > int num_simplexes_per_complex =
> > @toint(@arg("num_simplexes_per_complex"));
> >
> > app (file output, file log) cce_ua (file input1, file input2, int
> > number, file input3)
> > {
> > cce_ua @input1 @input2 @output number @input3 stdout=@filename(log)
> > stderr=@filename(log);
> > #located at _RUNDIR_/swat/cce_ua.sh
> > }
> >
> > file inSCE<single_file_mapper;file="SCE_Par.mat">;
> > file inrunswat<single_file_mapper;file="RunSwat.m">;
> >
> > foreach i in [1:num_complexes]
> > {
> > foreach j in [1:num_simplexes_per_complex]
> > {
> > # These files already exist in the local directory
> > file input<single_file_mapper; file=@strcat("simplex", i, "_", j,
> > ".mat")>;
> > # These files do not exist in the local directory at runtime
> > file output<single_file_mapper; file=@strcat("new_point", i, "_", j,
> > ".mat")>;
> > file log<single_file_mapper; file=@strcat("cce_ua", i, "_", j,
> > "_log.txt")>;
> >
> > (output, log) = cce_ua(inSCE, input, i*num_simplexes_per_complex + j,
> > inrunswat);
> > }
> > }
> >
> >
> > ---- The command line call to swift (broken up over multiple lines)
> > ---
> > swift -tc.file swift_files/executable_locations -sites.file
> > swift_files/sites.xml cce_ua.swift -num_complexes=2
> > -num_simplexes_per_complex=1
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120703/0c958524/attachment.html>

From wilde at mcs.anl.gov  Tue Jul  3 18:53:43 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 3 Jul 2012 18:53:43 -0500 (CDT)
Subject: [Swift-user] How does swift mapping to files that do not yet
 exist work?
In-Reply-To: <CAJgtAsouCAmQ7DrGAsSmdH3+7ELgNqSRdYE4nojy4GHRxX6tdQ@mail.gmail.com>
Message-ID: <1909058417.6829.1341359623604.JavaMail.root@zimbra.anl.gov>

Jonathan,

I tried running your script under a fairly old version of trunk, and it seems to work fine for me with the params you specified below. I used a simple shell script of echoes instead of a matlab script.

Swift should know that the file variable "output" is an output variable, and not expect it to exist before the app is run. And indeed, in my test of your script this works fine.

What version of Swift are you running? Can you post your swift log file, and the full swift output from stdout/err?

Is there some delta between what you posted and what you're actually running, perhaps?

I *did* get file-not-found errors for *input* files that were missing, until I provided all the expected input files, but the output file did not pre-exist and was created as expected:

com$ ls -l *.mat
-rw-r--r-- 1 wilde ci-users 12 Jul  3 18:36 SCE_Par.mat
-rw-r--r-- 1 wilde ci-users 47 Jul  3 18:41 new_point1_1.mat
-rw-r--r-- 1 wilde ci-users 47 Jul  3 18:41 new_point2_1.mat
-rw-r--r-- 1 wilde ci-users 20 Jul  3 18:38 simplex1_1.mat
-rw-r--r-- 1 wilde ci-users 20 Jul  3 18:41 simplex2_1.mat
com$ more new*.mat
::::::::::::::
new_point1_1.mat
::::::::::::::
this is the output file named new_point1_1.mat
::::::::::::::
new_point2_1.mat
::::::::::::::
this is the output file named new_point2_1.mat
com$ 


- Mike




----- Original Message -----
> From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: swift-user at ci.uchicago.edu
> Sent: Tuesday, July 3, 2012 5:24:19 PM
> Subject: Re: [Swift-user] How does swift mapping to files that do not yet exist work?
> Sorry, I should have specified this earlier, but I do not believe my
> cce_ua function was ever called. The first line of my cce_ua function
> is a statement that prints to stdout. The second and third lines open
> a file (via absolute path) and write to it. I have not observed either
> of those being executed. Furthermore, the following (matlab) code
> process the output argument:
> 
> output_point_location = argv(){3};
> 
> new_point = ....
> 
> save(output_point_location, 'new_point');
> 
> There is no other code that modifies the output_point_location
> variable. So shouldn't this work, given how swift passes files as
> command line arguments?
> 
> I think something else is the problem, but of course any solution
> would be great. Thanks!
> 
> Jonathan
> 
> 
> On Tue, Jul 3, 2012 at 5:11 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> Jonathan, from a cursory read of your script, I suspect that the Swift
> runtime is complaining that the app cce_ua did not produce the file
> new_point1_1.mat, which your output file mapping specified.
> 
> Given your output mapping, Swift is expecting new_point1_1.mat to
> exist in the temporary job directory in which your app was launched. I
> suspect your app (matlab script) did not create a file of this exact
> name.
> 
> You can specify the property "sitedir.keep=true" in your config file
> (e.g., -config cf on the command line) to tell Swift to retain these
> job directories on failure. They should be below the "workdirectory"
> that you specified in your sites.xml file.
> (I think...)
> 
> - Mike
> 
> 
> 
> 
> ----- Original Message -----
> > From: "Jonathan Margoliash" < jmargolpeople at gmail.com >
> > To: swift-user at ci.uchicago.edu
> > Sent: Tuesday, July 3, 2012 4:48:16 PM
> > Subject: [Swift-user] How does swift mapping to files that do not
> > yet exist work?
> > Hello swift user base,
> >
> > I have been having trouble with a bit of swift code for a while now,
> > and despite my hesitancy to ask other people to debug my code, I
> > believe my issue may be more of a lack of understanding of swift
> > than
> > a simple syntax error, so I have decided to send out this email.
> > Specifically, I have a single_file_mapper mapping to a file which
> > does
> > not yet exist when my code is run, in hopes that the file being
> > mapped
> > to will be written to by a an app call. However, when I try running
> > my
> > code, I get the error back from swift
> >
> > Execution failed:
> > File not found: filename (in this case new_point1_1.mat)
> >
> > This paradigm of referencing files that do not yet exist has worked
> > before in swift, so I do not know what the problem is here.
> > Hopefully
> > it is not a simple syntax bug. Anyway, thank you for your help,
> >
> > Jonathan
> >
> > ----- My Code: ----------
> >
> > #cce_ua.swift
> > type file;
> >
> > int num_complexes = @toint(@arg("num_complexes"));
> > int num_simplexes_per_complex =
> > @toint(@arg("num_simplexes_per_complex"));
> >
> > app (file output, file log) cce_ua (file input1, file input2, int
> > number, file input3)
> > {
> > cce_ua @input1 @input2 @output number @input3 stdout=@filename(log)
> > stderr=@filename(log);
> > #located at _RUNDIR_/swat/cce_ua.sh
> > }
> >
> > file inSCE<single_file_mapper;file="SCE_Par.mat">;
> > file inrunswat<single_file_mapper;file="RunSwat.m">;
> >
> > foreach i in [1:num_complexes]
> > {
> > foreach j in [1:num_simplexes_per_complex]
> > {
> > # These files already exist in the local directory
> > file input<single_file_mapper; file=@strcat("simplex", i, "_", j,
> > ".mat")>;
> > # These files do not exist in the local directory at runtime
> > file output<single_file_mapper; file=@strcat("new_point", i, "_", j,
> > ".mat")>;
> > file log<single_file_mapper; file=@strcat("cce_ua", i, "_", j,
> > "_log.txt")>;
> >
> > (output, log) = cce_ua(inSCE, input, i*num_simplexes_per_complex +
> > j,
> > inrunswat);
> > }
> > }
> >
> >
> > ---- The command line call to swift (broken up over multiple lines)
> > ---
> > swift -tc.file swift_files/executable_locations -sites.file
> > swift_files/sites.xml cce_ua.swift -num_complexes=2
> > -num_simplexes_per_complex=1
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> --
> Michael Wilde
> Computation Institute, University of Chicago
> Mathematics and Computer Science Division
> Argonne National Laboratory

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jmargolpeople at gmail.com  Fri Jul  6 11:26:53 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Fri, 6 Jul 2012 11:26:53 -0500
Subject: [Swift-user] Swift crashing,
	says it requires a config property which is not documente
Message-ID: <CAJgtAsp=T2HwokQzknqOc68H+v75Q4uSn8irSYyhnrUPsqKjtw@mail.gmail.com>

Hello swift user base,

Earlier today I tried running my swift scripts on the trunk version of
swift (downloading and installing it as instructed under the "Development
Version" heading of the swift downloads page). I added swift to my path,
and then tried running my scripts, which produced the following error
message:

Execution failed:
Swift config property "use.wrapper.staging" not found in Swift
configuration [swift_files/config]

There was no other output of the swift program. I tried looking in the
online documentation for use.wrapper.staging or anything that looked like
it, but could not find the setting. Can you explain to me what this
configuration setting is supposed to do, why swift is crashing without it,
and if I should add it to my config file? Thanks,

Jonathan

btw - swift_files/config is the location I store the config folder relative
to the swift script being run. It is not built in to the error message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120706/40a97546/attachment.html>

From wilde at mcs.anl.gov  Fri Jul  6 13:21:25 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 6 Jul 2012 13:21:25 -0500 (CDT)
Subject: [Swift-user] Swift crashing,
 says it requires a config property which is not documente
In-Reply-To: <CAJgtAsp=T2HwokQzknqOc68H+v75Q4uSn8irSYyhnrUPsqKjtw@mail.gmail.com>
Message-ID: <264864973.9799.1341598885984.JavaMail.root@zimbra.anl.gov>

Jonathan, sorry, this should be documented (its a new property), but for now, just add "use.wrapper.staging=false" to your -config file when using trunk.

- Mike

----- Original Message -----
> From: "Jonathan Margoliash" <jmargolpeople at gmail.com>
> To: swift-user at ci.uchicago.edu
> Sent: Friday, July 6, 2012 11:26:53 AM
> Subject: [Swift-user] Swift crashing, says it requires a config property which is not documente
> Hello swift user base,
> 
> 
> Earlier today I tried running my swift scripts on the trunk version of
> swift (downloading and installing it as instructed under the
> "Development Version" heading of the swift downloads page). I added
> swift to my path, and then tried running my scripts, which produced
> the following error message:
> 
> 
> 
> Execution failed:
> Swift config property "use.wrapper.staging" not found in Swift
> configuration [swift_files/config]
> 
> 
> There was no other output of the swift program. I tried looking in the
> online documentation for use.wrapper.staging or anything that looked
> like it, but could not find the setting. Can you explain to me what
> this configuration setting is supposed to do, why swift is crashing
> without it, and if I should add it to my config file? Thanks,
> 
> 
> Jonathan
> 
> 
> btw - swift_files/config is the location I store the config folder
> relative to the swift script being run. It is not built in to the
> error message.
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From Khushbu.Agarwal at pnnl.gov  Tue Jul 10 15:55:58 2012
From: Khushbu.Agarwal at pnnl.gov (Agarwal, Khushbu)
Date: Tue, 10 Jul 2012 13:55:58 -0700
Subject: [Swift-user] moving files during runtime
Message-ID: <D0B71746ADD17848B85A4808D5A61667012AE3D90ABA@EMAIL05.pnl.gov>

Hi,

I have generic question about moving files around during swift runtime. We have an app procedure being invoked from swift, across multiple iterations. In the shell script for the app, I actually do a move of the file that was produced in my previous iteration to my current iteration. However when Swift runs, I just see lot of links.  Basically file in iteration 4  has a symbolic link to file in iteration 3, 3 to 2 and so on. This works ok for couple of iterations, but after a while system seem to fail with message " unable to follow links". Since we are trying to run over 100's of iterations, I would like to know if there is a way to tell swift to   just copy or move the file than to create symbolic links.

Thanks much,
-Khushbu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120710/dccf9352/attachment.html>

From wilde at mcs.anl.gov  Tue Jul 10 21:44:43 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 10 Jul 2012 21:44:43 -0500 (CDT)
Subject: [Swift-user] moving files during runtime
In-Reply-To: <D0B71746ADD17848B85A4808D5A61667012AE3D90ABA@EMAIL05.pnl.gov>
Message-ID: <1614084455.15298.1341974683371.JavaMail.root@zimbra.anl.gov>

Hi Khushbu,

> ...In the shell script for the app, I actually do a
> move of the file that was produced in my previous iteration to my
> current iteration. However when Swift runs, I just see lot of links.
> Basically file in iteration 4 has a symbolic link to file in iteration
> 3, 3 to 2 and so on. This works ok for couple of iterations, but after
> a while system seem to fail with message ? unable to follow links?.
> Since we are trying to run over 100?s of iterations, I would like to
> know if there is a way to tell swift to just copy or move the file
> than to create symbolic links.

I'm assuming that since each iteration of this process depends on the prior iteration, they all run serially.

If thats the case, you can use either a foreach loop or an iterate statement, and just let Swift do all the copying in a natural way:

type file;
file f[];
file first<"initial.data">;

f[0] = first;
foreach i in [1:100] {
  f[i] = myApp(f[i-1]);
}

(I didnt test this, but something similar should work for your application)

Whats happening in your current case is that your app script is "breaking the rules" by trying to preserve data objects from the Swift pre-job "sandbox" directory (as explained in User Guide sections 5.1 and 5.2:
  http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_mapping_of_app_semantics_into_unix_process_execution_semantics

and 

http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_how_swift_implements_the_site_execution_model

I could explain a few ways to make your current approach work, but I think the approach above is much more natural and also safer and less brittle - less vulnerable to breakage if we change the Swift runtime model. (For example, you could do the cp in your app script rather than an mv, or you could Swift to do cp's rather than links by specifying the "scratch" tag in your sites.xml entry. But I didn't tell you either of these things because they are not good practice ;)

- Mike







From wilde at mcs.anl.gov  Tue Jul 10 22:04:51 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Tue, 10 Jul 2012 22:04:51 -0500 (CDT)
Subject: [Swift-user] moving files during runtime
In-Reply-To: <1614084455.15298.1341974683371.JavaMail.root@zimbra.anl.gov>
Message-ID: <1562715601.15304.1341975891405.JavaMail.root@zimbra.anl.gov>

Khushbu, this example works for me:

$ cat serial.swift

type file;

app (file o) incfile (file i)
{
  sh "-c" "echo $(($(cat) + 1))" stdin=@i stdout=@o;
}
  
file f[];
file first<"f.0">;

f[0] = first;
foreach i in [1:10] {
  file newf<single_file_mapper; file=@strcat("f.",i)>;
  newf = incfile(f[i-1]);
  f[i] = newf;
}

$ cat tsh
localhost sh /bin/sh

$ swift -tc.file tsh serial.swift
no sites file specified, setting to default: /software/swift-0.93-all/etc/sites.xml
Swift 0.93 swift-r5483 cog-r3339

RunID: 20120711-0259-mkjdrq1g
Progress:  time: Wed, 11 Jul 2012 03:00:00 +0000
Final status:  time: Wed, 11 Jul 2012 03:00:01 +0000  Finished successfully:10

$ more f.*
::::::::::::::
f.0
::::::::::::::
100
::::::::::::::
f.1
::::::::::::::
101
::::::::::::::
f.10
::::::::::::::
110
::::::::::::::
f.2
::::::::::::::
102
::::::::::::::
f.3
::::::::::::::
103
::::::::::::::
f.4
::::::::::::::
104
::::::::::::::
f.5
::::::::::::::
105
::::::::::::::
f.6
::::::::::::::
106
::::::::::::::
f.7
::::::::::::::
107
::::::::::::::
f.8
::::::::::::::
108
::::::::::::::
f.9
::::::::::::::
109
$ 


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Khushbu Agarwal" <Khushbu.Agarwal at pnnl.gov>
> Cc: "Karen L Schuchardt" <Karen.Schuchardt at pnnl.gov>, swift-user at ci.uchicago.edu
> Sent: Tuesday, July 10, 2012 9:44:43 PM
> Subject: Re: [Swift-user] moving files during runtime
> Hi Khushbu,
> 
> > ...In the shell script for the app, I actually do a
> > move of the file that was produced in my previous iteration to my
> > current iteration. However when Swift runs, I just see lot of links.
> > Basically file in iteration 4 has a symbolic link to file in
> > iteration
> > 3, 3 to 2 and so on. This works ok for couple of iterations, but
> > after
> > a while system seem to fail with message ? unable to follow links?.
> > Since we are trying to run over 100?s of iterations, I would like to
> > know if there is a way to tell swift to just copy or move the file
> > than to create symbolic links.
> 
> I'm assuming that since each iteration of this process depends on the
> prior iteration, they all run serially.
> 
> If thats the case, you can use either a foreach loop or an iterate
> statement, and just let Swift do all the copying in a natural way:
> 
> type file;
> file f[];
> file first<"initial.data">;
> 
> f[0] = first;
> foreach i in [1:100] {
> f[i] = myApp(f[i-1]);
> }
> 
> (I didnt test this, but something similar should work for your
> application)
> 
> Whats happening in your current case is that your app script is
> "breaking the rules" by trying to preserve data objects from the Swift
> pre-job "sandbox" directory (as explained in User Guide sections 5.1
> and 5.2:
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_mapping_of_app_semantics_into_unix_process_execution_semantics
> 
> and
> 
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_how_swift_implements_the_site_execution_model
> 
> I could explain a few ways to make your current approach work, but I
> think the approach above is much more natural and also safer and less
> brittle - less vulnerable to breakage if we change the Swift runtime
> model. (For example, you could do the cp in your app script rather
> than an mv, or you could Swift to do cp's rather than links by
> specifying the "scratch" tag in your sites.xml entry. But I didn't
> tell you either of these things because they are not good practice ;)
> 
> - Mike
> 
> 
> 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From jmargolpeople at gmail.com  Fri Jul 13 17:32:14 2012
From: jmargolpeople at gmail.com (Jonathan Margoliash)
Date: Fri, 13 Jul 2012 17:32:14 -0500
Subject: [Swift-user] Quick swift related question
Message-ID: <CAJgtAso4CN0R8Q97Jg8oc7_B00ggpFiZ_korRu-Efc10HXSxxw@mail.gmail.com>

When running a script called by swift, is it correct to assume that it
should not reference any files not explicitly passed to it as arguments by
swift? Also, what is the best way of passing large quantities of files to a
script from swift? Can swift pass directories wholesale, or should they be
tarballed first? And if the script needs many files in different locations,
must they all be passed by name, or is there some more convenient method of
passing them?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120713/6b915529/attachment.html>

From robinweiss at uchicago.edu  Mon Jul 16 10:43:28 2012
From: robinweiss at uchicago.edu (Robin Weiss)
Date: Mon, 16 Jul 2012 15:43:28 +0000
Subject: [Swift-user] using dual filesys_mappers
Message-ID: <CC29A0D1.4C1D%robinweiss@uchicago.edu>

Hello Swifters,


I have a question about using two filesys_mappers and the foreach construct.  I have attached the offending .swift script I am working with for reference.  Here's the gist of what i'm trying to do and what appears to be happening instead:


I have a program called 'boot' that takes as command line arguments 4 file paths (log, out, data, and meta), and a number of other params (all numerical and seem to be getting passed in correctly, so no worries there).  'log' and 'out' are output files and 'data' and 'mata' are input files (located in directories called 'out' and 'in' respectively).  The problem i'm having is with getting the correct values for 'data' and 'meta' passed in to my app call.  I have an app section called processData that invokes boot.  I will be assuming the the directory 'in' contains identically named data and meta files that differ only in their suffix ('.dat' or '.meta', respectively).  This may or may not be safe, but for now it is fine and I'll cross that bridge when I come to it.  Here's the relevant snippet from my script:



app (clusFile out) processData (dataFile data, metaFile meta, logFile log){


        boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;


}


dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;

metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


this configuration causes processData to be invoked as:


out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat, out/logFile.0.log);


if i switch around the oder of the filesys_mappers so that the snippet reads:


metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;

dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


the app invocation is called as:


out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta, out/logFile.0.log);


I guess the real question is this: what is the most appropriate way in swift to pass into a app invocation two corresponding input files?  Ideally, it would be something like 'foreach file1,file2,i in inputFiles[][] { ? } but that doesn't really make too much sense either.


Anyway, any ideas would be appreciated.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120716/612c1986/attachment.html>

From robinweiss at uchicago.edu  Mon Jul 16 12:34:23 2012
From: robinweiss at uchicago.edu (Robin Weiss)
Date: Mon, 16 Jul 2012 17:34:23 +0000
Subject: [Swift-user] using dual filesys_mappers
In-Reply-To: <CC29A0D1.4C1D%robinweiss@uchicago.edu>
Message-ID: <CC29B97C.4C20%robinweiss@uchicago.edu>

After a bit more playing, i have this working now.  I also realized that 'log' is really an output and shouldn't be passed into my app as an input (doh!).  The way I'm doing it now is as follows:


app (clusFile clus, logFile log) processData (dataFile data, metaFile meta){


boot "-log" @log "-results" @clus "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;

}


metaFile metaFiles[] <filesys_mapper;location="in",pattern="*.meta">;

dataFile dataFiles[] <filesys_mapper;location="in",pattern="*.dat">;


foreach f,i in dataFiles {


clusFile clus<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2clus">;

logFile log<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2log">;


(clus,log) = processData(f, metaFiles[i]);


}



This script is now mapping the .meta and .dat files from the 'in' directory with the pattern option of filesys_mapper.  Perhaps i'm missing something about what the suffix option does.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030

From: Robin Weiss <robinweiss at uchicago.edu<mailto:robinweiss at uchicago.edu>>
Date: Mon, 16 Jul 2012 10:43:29 -0500
To: <swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>>
Subject: using dual filesys_mappers


Hello Swifters,


I have a question about using two filesys_mappers and the foreach construct.  I have attached the offending .swift script I am working with for reference.  Here's the gist of what i'm trying to do and what appears to be happening instead:


I have a program called 'boot' that takes as command line arguments 4 file paths (log, out, data, and meta), and a number of other params (all numerical and seem to be getting passed in correctly, so no worries there).  'log' and 'out' are output files and 'data' and 'mata' are input files (located in directories called 'out' and 'in' respectively).  The problem i'm having is with getting the correct values for 'data' and 'meta' passed in to my app call.  I have an app section called processData that invokes boot.  I will be assuming the the directory 'in' contains identically named data and meta files that differ only in their suffix ('.dat' or '.meta', respectively).  This may or may not be safe, but for now it is fine and I'll cross that bridge when I come to it.  Here's the relevant snippet from my script:



app (clusFile out) processData (dataFile data, metaFile meta, logFile log){


        boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;


}


dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;

metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


this configuration causes processData to be invoked as:


out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat, out/logFile.0.log);


if i switch around the oder of the filesys_mappers so that the snippet reads:


metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;

dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


the app invocation is called as:


out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta, out/logFile.0.log);


I guess the real question is this: what is the most appropriate way in swift to pass into a app invocation two corresponding input files?  Ideally, it would be something like 'foreach file1,file2,i in inputFiles[][] { ? } but that doesn't really make too much sense either.


Anyway, any ideas would be appreciated.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120716/28ef5d71/attachment.html>

From wilde at mcs.anl.gov  Wed Jul 18 16:48:43 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Wed, 18 Jul 2012 16:48:43 -0500 (CDT)
Subject: [Swift-user] using dual filesys_mappers
In-Reply-To: <CC29B97C.4C20%robinweiss@uchicago.edu>
Message-ID: <193137093.26014.1342648123793.JavaMail.root@zimbra.anl.gov>

Robin, here's an example of using the suffix option for the filesys_mapper:

login2$ ls *.pcap
pc00.pcap  pc01.pcap  pc02.pcap  pc03.pcap  pcaa.pcap  pczz.pcap
login2$ cat filesysmapper.swift
type pcapfile;  

pcapfile pcapfiles[]<filesys_mapper; prefix="pc", suffix=".pcap">;

foreach pf,i in pcapfiles {
#  trace(i, at filename(pf));
}

iterate i {
  trace(i, @filename(pcapfiles[i]));
} until (i==@length(pcapfiles));

login2$ swift filesysmapper.swift
no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml
Swift trunk swift-r5781 cog-r3368 (cog modified locally)

RunID: 20120718-1646-fm7zxfka
 (input): found 6 files
Progress:  time: Wed, 18 Jul 2012 16:46:12 -0500
SwiftScript trace: 0, pczz.pcap
SwiftScript trace: 1, pc00.pcap
SwiftScript trace: 2, pc01.pcap
SwiftScript trace: 3, pc03.pcap
SwiftScript trace: 4, pcaa.pcap
SwiftScript trace: 5, pc02.pcap
Final status: Wed, 18 Jul 2012 16:46:12 -0500
login2$ 

I need to go back to your prior email to see if you discovered a problem with it.

- Mike

----- Original Message -----
> From: "Robin Weiss" <robinweiss at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Monday, July 16, 2012 12:34:23 PM
> Subject: Re: [Swift-user] using dual filesys_mappers
> After a bit more playing, i have this working now. I also realized
> that 'log' is really an output and shouldn't be passed into my app as
> an input (doh!). The way I'm doing it now is as follows:
> 
> 
> 
> 
> app (clusFile clus, logFile log) processData (dataFile data, metaFile
> meta){
> 
> 
> 
> 
> boot "-log" @log "-results" @clus "-meta" @meta "-data" @data "-kmin"
> kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations"
> maxiterations "-maxtries" maxtries;
> 
> 
> 
> }
> 
> 
> 
> 
> metaFile metaFiles[] <filesys_mapper;location="in",pattern="*.meta">;
> 
> dataFile dataFiles[] <filesys_mapper;location="in",pattern="*.dat">;
> 
> 
> 
> 
> foreach f,i in dataFiles {
> 
> 
> 
> 
> clusFile
> clus<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2clus">;
> 
> logFile
> log<regexp_mapper;source=@f,match="(in)/(.*)dat",transform="out/\\2log">;
> 
> 
> 
> 
> (clus,log) = processData(f, metaFiles[i]);
> 
> 
> 
> 
> }
> 
> 
> 
> 
> 
> 
> 
> This script is now mapping the .meta and .dat files from the 'in'
> directory with the pattern option of filesys_mapper. Perhaps i'm
> missing something about what the suffix option does.
> 
> 
> 
> 
> Cheers,
> 
> Robin
> 
> 
> 
> 
> 
> --
> 
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
> 
> 
> From: Robin Weiss < robinweiss at uchicago.edu >
> Date: Mon, 16 Jul 2012 10:43:29 -0500
> To: < swift-user at ci.uchicago.edu >
> Subject: using dual filesys_mappers
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Hello Swifters,
> 
> 
> 
> 
> I have a question about using two filesys_mappers and the foreach
> construct. I have attached the offending .swift script I am working
> with for reference. Here's the gist of what i'm trying to do and what
> appears to be happening instead:
> 
> 
> 
> 
> I have a program called 'boot' that takes as command line arguments 4
> file paths (log, out, data, and meta), and a number of other params
> (all numerical and seem to be getting passed in correctly, so no
> worries there). 'log' and 'out' are output files and 'data' and 'mata'
> are input files (located in directories called 'out' and 'in'
> respectively). The problem i'm having is with getting the correct
> values for 'data' and 'meta' passed in to my app call. I have an app
> section called processData that invokes boot. I will be assuming the
> the directory 'in' contains identically named data and meta files that
> differ only in their suffix ('.dat' or '.meta', respectively). This
> may or may not be safe, but for now it is fine and I'll cross that
> bridge when I come to it. Here's the relevant snippet from my script:
> 
> 
> 
> 
> 
> 
> 
> app (clusFile out) processData (dataFile data, metaFile meta, logFile
> log){
> 
> 
> 
> 
> boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin"
> kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations"
> maxiterations "-maxtries" maxtries;
> 
> 
> 
> 
> }
> 
> 
> 
> 
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
> 
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
> 
> 
> 
> 
> foreach f,i in dataFiles {
> 
> 
> 
> 
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
> 
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
> 
> o = processData(f, metaFiles[i], l);
> 
> 
> 
> 
> }
> 
> 
> 
> 
> this configuration causes processData to be invoked as:
> 
> 
> 
> 
> out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat,
> out/logFile.0.log);
> 
> 
> 
> 
> if i switch around the oder of the filesys_mappers so that the snippet
> reads:
> 
> 
> 
> 
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
> 
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
> 
> 
> 
> 
> foreach f,i in dataFiles {
> 
> 
> 
> 
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
> 
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
> 
> o = processData(f, metaFiles[i], l);
> 
> 
> 
> 
> }
> 
> 
> 
> 
> the app invocation is called as:
> 
> 
> 
> 
> out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta,
> out/logFile.0.log);
> 
> 
> 
> 
> I guess the real question is this: what is the most appropriate way in
> swift to pass into a app invocation two corresponding input files?
> Ideally, it would be something like 'foreach file1,file2,i in
> inputFiles[][] { ? } but that doesn't really make too much sense
> either.
> 
> 
> 
> 
> Anyway, any ideas would be appreciated.
> 
> 
> 
> 
> Cheers,
> 
> Robin
> 
> 
> 
> 
> 
> --
> 
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From robinweiss at uchicago.edu  Fri Jul 20 11:50:39 2012
From: robinweiss at uchicago.edu (Robin Weiss)
Date: Fri, 20 Jul 2012 16:50:39 +0000
Subject: [Swift-user] using dual filesys_mappers
In-Reply-To: <CC29A0D1.4C1D%robinweiss@uchicago.edu>
Message-ID: <CC2EF49D.4CFA%robinweiss@uchicago.edu>

So it turns out this was a PEBKAC issue.  While I think I know why swift is doing what it's doing, this behavior is a little tricky to diagnose and I was totally looking at the problem but not seeing it.  Let me explain.

Here is the offending swift script and the output from running it.  The directory 'in' contains a number of .dat files and corresponding .meta files (name differs only in extension).  The expected output in each outFile file should be "-meta fileX.meta -dat fileX.dat", "-meta fileY.meta -dat fileY.dat", etc.


type datFile;
type metaFile;
type outFile;

app (outFile out) testApp (datFile dat, metaFile meta){

echo "-meta" @meta "-dat" @dat stdout=@out;

}

datFile datFiles[] <filesys_mapper;location="in",sufix="dat">;
metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;

foreach f,i in datFiles {

outFile out <concurrent_mapper;prefix="out", suffix=".txt">;
out = testApp(f, metaFiles[i]);

}


[robinweiss at midway037 bad_swift]$ pwd
/home/robinweiss/bad_swift
[robinweiss at midway037 bad_swift]$ cd in
[robinweiss at midway037 in]$ ls
file0.dat  file0.meta  file1.dat  file1.meta  file2.dat  file2.meta  file3.dat  file3.meta
[robinweiss at midway037 in]$ cd ..
[robinweiss at midway037 bad_swift]$ ./runLocal.sh
Swift 0.93 swift-r5483 cog-r3339

RunID: 20120720-1623-cpgd5xr9
 (input): found 8 files
 (input): found 8 files
Progress:  time: Fri, 20 Jul 2012 16:23:48 +0000  Initializing:1
Final status:  time: Fri, 20 Jul 2012 16:23:49 +0000  Finished successfully:8
[robinweiss at midway037 bad_swift]$ cd _concurrent/
[robinweiss at midway037 _concurrent]$ ls
out-4-0.txt  out-4-1.txt  out-4-2.txt  out-4-3.txt  out-4-4.txt  out-4-5.txt  out-4-6.txt  out-4-7.txt
[robinweiss at midway037 _concurrent]$ cat *
-meta in/file0.meta -dat in/file0.meta
-meta in/file0.dat -dat in/file0.dat
-meta in/file1.meta -dat in/file1.meta
-meta in/file1.dat -dat in/file1.dat
-meta in/file2.dat -dat in/file2.dat
-meta in/file3.dat -dat in/file3.dat
-meta in/file3.meta -dat in/file3.meta
-meta in/file2.meta -dat in/file2.meta


Can you spot the problem in the script?  Hint: "suffix" has two f's, not one.  Because the parameter 'sufix' is meaningless to filesys_mapper it ends up just mapping everything in 'location'.  It would seem mappers will allow you to put in pretty much any garbage you want inside the < >'s so long as its an assignment.  For example:

datFile datFiles[] <filesys_mapper;location="in",suffix="dat",foo="bar",biz="baz",garbage="more garbage">;

works just fine also.


And now for the good run:

type datFile;
type metaFile;
type outFile;

app (outFile out) testApp (datFile dat, metaFile meta){

echo "-meta" @meta "-dat" @dat stdout=@out;

}

datFile datFiles[] <filesys_mapper;location="in",suffix="dat">;
metaFile metaFiles[] <filesys_mapper;location="in",suffix="meta">;

foreach f,i in datFiles {

outFile out <concurrent_mapper;prefix="out", suffix=".txt">;
out = testApp(f, metaFiles[i]);

}

[robinweiss at midway037 bad_swift]$ pwd
/home/robinweiss/bad_swift
[robinweiss at midway037 bad_swift]$ cd in
[robinweiss at midway037 in]$ ls
file0.dat  file0.meta  file1.dat  file1.meta  file2.dat  file2.meta  file3.dat  file3.meta
[robinweiss at midway037 in]$ cd ..
[robinweiss at midway037 bad_swift]$ ./runLocal.sh
Swift 0.93 swift-r5483 cog-r3339

RunID: 20120720-1626-ou1juqf5
 (input): found 4 files
 (input): found 4 files
Progress:  time: Fri, 20 Jul 2012 16:26:51 +0000  Initializing:1
Final status:  time: Fri, 20 Jul 2012 16:26:51 +0000  Finished successfully:4
[robinweiss at midway037 bad_swift]$ cd _concurrent/
[robinweiss at midway037 _concurrent]$ ls
out-4-0.txt  out-4-1.txt  out-4-2.txt  out-4-3.txt
[robinweiss at midway037 _concurrent]$ cat *
-meta in/file0.meta -dat in/file0.dat
-meta in/file1.meta -dat in/file1.dat
-meta in/file3.meta -dat in/file2.dat
-meta in/file2.meta -dat in/file3.dat
[robinweiss at midway037 _concurrent]$


So like I said, it turns out this whole issue boils down to a typo.  I would suggest that swift should throw up some sort of warning if you pass something to one of the pre-defined mappers that is not defined as one of its parameters.  I would have expected swift to complain that "sufix" is an unknown parameter of filesys_mapper.

Cheers,
Robin

--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030

From: Robin Weiss <robinweiss at uchicago.edu<mailto:robinweiss at uchicago.edu>>
Date: Mon, 16 Jul 2012 10:43:29 -0500
To: <swift-user at ci.uchicago.edu<mailto:swift-user at ci.uchicago.edu>>
Subject: using dual filesys_mappers


Hello Swifters,


I have a question about using two filesys_mappers and the foreach construct.  I have attached the offending .swift script I am working with for reference.  Here's the gist of what i'm trying to do and what appears to be happening instead:


I have a program called 'boot' that takes as command line arguments 4 file paths (log, out, data, and meta), and a number of other params (all numerical and seem to be getting passed in correctly, so no worries there).  'log' and 'out' are output files and 'data' and 'mata' are input files (located in directories called 'out' and 'in' respectively).  The problem i'm having is with getting the correct values for 'data' and 'meta' passed in to my app call.  I have an app section called processData that invokes boot.  I will be assuming the the directory 'in' contains identically named data and meta files that differ only in their suffix ('.dat' or '.meta', respectively).  This may or may not be safe, but for now it is fine and I'll cross that bridge when I come to it.  Here's the relevant snippet from my script:



app (clusFile out) processData (dataFile data, metaFile meta, logFile log){


        boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin" kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations" maxiterations "-maxtries" maxtries;


}


dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;

metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


this configuration causes processData to be invoked as:


out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat, out/logFile.0.log);


if i switch around the oder of the filesys_mappers so that the snippet reads:


metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;

dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;


foreach f,i in dataFiles {


        clusFile o<single_file_mapper; location="out", file=@strcat("out/clusFile.",i,".clus")>;

        logFile l<single_file_mapper; location="out", file=@strcat("out/logFile.",i,".log")>;

        o = processData(f, metaFiles[i], l);


}


the app invocation is called as:


out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta, out/logFile.0.log);


I guess the real question is this: what is the most appropriate way in swift to pass into a app invocation two corresponding input files?  Ideally, it would be something like 'foreach file1,file2,i in inputFiles[][] { ? } but that doesn't really make too much sense either.


Anyway, any ideas would be appreciated.


Cheers,

Robin


--
Robin M. Weiss
Research Programmer
Research Computing Center
The University of Chicago
6030 S. Ellis Ave., Suite 289C
Chicago, IL 60637
robinweiss at uchicago.edu<mailto:labello at uchicago.edu>
773.702.9030
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120720/7df810d8/attachment.html>

From wilde at mcs.anl.gov  Fri Jul 20 13:48:40 2012
From: wilde at mcs.anl.gov (Michael Wilde)
Date: Fri, 20 Jul 2012 13:48:40 -0500 (CDT)
Subject: [Swift-user] using dual filesys_mappers
In-Reply-To: <CC2EF49D.4CFA%robinweiss@uchicago.edu>
Message-ID: <695479345.29090.1342810120667.JavaMail.root@zimbra.anl.gov>

Thanks, Robin. I agree, Swift should have warned you. Ive entered this into bugzilla:

  https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=801

- Mike


----- Original Message -----
> From: "Robin Weiss" <robinweiss at uchicago.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Friday, July 20, 2012 11:50:39 AM
> Subject: Re: [Swift-user] using dual filesys_mappers
> So it turns out this was a PEBKAC issue. While I think I know why
> swift is doing what it's doing, this behavior is a little tricky to
> diagnose and I was totally looking at the problem but not seeing it.
> Let me explain.
> 
> 
> Here is the offending swift script and the output from running it. The
> directory 'in' contains a number of .dat files and corresponding .meta
> files (name differs only in extension). The expected output in each
> outFile file should be "-meta fileX.meta -dat fileX.dat", "-meta
> fileY.meta -dat fileY.dat", etc.
> 
> 
> 
> 
> type datFile;
> type metaFile;
> type outFile;
> 
> 
> app (outFile out) testApp (datFile dat, metaFile meta){
> 
> 
> echo "-meta" @meta "-dat" @dat stdout=@out;
> 
> 
> }
> 
> 
> datFile datFiles[] <filesys_mapper;location="in",sufix="dat">;
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
> 
> 
> foreach f,i in datFiles {
> 
> 
> outFile out <concurrent_mapper;prefix="out", suffix=".txt">;
> 
> out = testApp(f, metaFiles[i]);
> 
> }
> 
> 
> 
> 
> [robinweiss at midway037 bad_swift]$ pwd
> /home/robinweiss/bad_swift
> [robinweiss at midway037 bad_swift]$ cd in
> [robinweiss at midway037 in]$ ls
> file0.dat file0.meta file1.dat file1.meta file2.dat file2.meta
> file3.dat file3.meta
> [robinweiss at midway037 in]$ cd ..
> [robinweiss at midway037 bad_swift]$ ./runLocal.sh
> Swift 0.93 swift-r5483 cog-r3339
> 
> 
> RunID: 20120720-1623-cpgd5xr9
> (input): found 8 files
> (input): found 8 files
> Progress: time: Fri, 20 Jul 2012 16:23:48 +0000 Initializing:1
> Final status: time: Fri, 20 Jul 2012 16:23:49 +0000 Finished
> successfully:8
> [robinweiss at midway037 bad_swift]$ cd _concurrent/
> [robinweiss at midway037 _concurrent]$ ls
> out-4-0.txt out-4-1.txt out-4-2.txt out-4-3.txt out-4-4.txt
> out-4-5.txt out-4-6.txt out-4-7.txt
> [robinweiss at midway037 _concurrent]$ cat *
> -meta in/file0.meta -dat in/file0.meta
> -meta in/file0.dat -dat in/file0.dat
> -meta in/file1.meta -dat in/file1.meta
> -meta in/file1.dat -dat in/file1.dat
> -meta in/file2.dat -dat in/file2.dat
> -meta in/file3.dat -dat in/file3.dat
> -meta in/file3.meta -dat in/file3.meta
> -meta in/file2.meta -dat in/file2.meta
> 
> 
> 
> 
> Can you spot the problem in the script? Hint: "suffix" has two f's,
> not one. Because the parameter 'sufix' is meaningless to
> filesys_mapper it ends up just mapping everything in 'location'. It
> would seem mappers will allow you to put in pretty much any garbage
> you want inside the < >'s so long as its an assignment. For example:
> 
> 
> datFile datFiles[]
> <filesys_mapper;location="in",suffix="dat",foo="bar",biz="baz",garbage="more
> garbage">;
> 
> 
> works just fine also.
> 
> 
> 
> 
> And now for the good run:
> 
> 
> type datFile;
> type metaFile;
> type outFile;
> 
> 
> app (outFile out) testApp (datFile dat, metaFile meta){
> 
> 
> echo "-meta" @meta "-dat" @dat stdout=@out;
> 
> 
> }
> 
> 
> datFile datFiles[] <filesys_mapper;location="in",suffix="dat">;
> metaFile metaFiles[] <filesys_mapper;location="in",suffix="meta">;
> 
> 
> foreach f,i in datFiles {
> 
> 
> outFile out <concurrent_mapper;prefix="out", suffix=".txt">;
> 
> out = testApp(f, metaFiles[i]);
> 
> }
> 
> 
> [robinweiss at midway037 bad_swift]$ pwd
> /home/robinweiss/bad_swift
> [robinweiss at midway037 bad_swift]$ cd in
> [robinweiss at midway037 in]$ ls
> file0.dat file0.meta file1.dat file1.meta file2.dat file2.meta
> file3.dat file3.meta
> [robinweiss at midway037 in]$ cd ..
> [robinweiss at midway037 bad_swift]$ ./runLocal.sh
> Swift 0.93 swift-r5483 cog-r3339
> 
> 
> RunID: 20120720-1626-ou1juqf5
> (input): found 4 files
> (input): found 4 files
> Progress: time: Fri, 20 Jul 2012 16:26:51 +0000 Initializing:1
> Final status: time: Fri, 20 Jul 2012 16:26:51 +0000 Finished
> successfully:4
> [robinweiss at midway037 bad_swift]$ cd _concurrent/
> [robinweiss at midway037 _concurrent]$ ls
> out-4-0.txt out-4-1.txt out-4-2.txt out-4-3.txt
> [robinweiss at midway037 _concurrent]$ cat *
> -meta in/file0.meta -dat in/file0.dat
> -meta in/file1.meta -dat in/file1.dat
> -meta in/file3.meta -dat in/file2.dat
> -meta in/file2.meta -dat in/file3.dat
> [robinweiss at midway037 _concurrent]$
> 
> 
> 
> 
> So like I said, it turns out this whole issue boils down to a typo. I
> would suggest that swift should throw up some sort of warning if you
> pass something to one of the pre-defined mappers that is not defined
> as one of its parameters. I would have expected swift to complain that
> "sufix" is an unknown parameter of filesys_mapper.
> 
> 
> Cheers,
> Robin
> 
> 
> 
> 
> --
> 
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
> 
> 
> From: Robin Weiss < robinweiss at uchicago.edu >
> Date: Mon, 16 Jul 2012 10:43:29 -0500
> To: < swift-user at ci.uchicago.edu >
> Subject: using dual filesys_mappers
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Hello Swifters,
> 
> 
> 
> 
> I have a question about using two filesys_mappers and the foreach
> construct. I have attached the offending .swift script I am working
> with for reference. Here's the gist of what i'm trying to do and what
> appears to be happening instead:
> 
> 
> 
> 
> I have a program called 'boot' that takes as command line arguments 4
> file paths (log, out, data, and meta), and a number of other params
> (all numerical and seem to be getting passed in correctly, so no
> worries there). 'log' and 'out' are output files and 'data' and 'mata'
> are input files (located in directories called 'out' and 'in'
> respectively). The problem i'm having is with getting the correct
> values for 'data' and 'meta' passed in to my app call. I have an app
> section called processData that invokes boot. I will be assuming the
> the directory 'in' contains identically named data and meta files that
> differ only in their suffix ('.dat' or '.meta', respectively). This
> may or may not be safe, but for now it is fine and I'll cross that
> bridge when I come to it. Here's the relevant snippet from my script:
> 
> 
> 
> 
> 
> 
> 
> app (clusFile out) processData (dataFile data, metaFile meta, logFile
> log){
> 
> 
> 
> 
> boot "-log" @log "-results" @out "-meta" @meta "-data" @data "-kmin"
> kmin "-kmax" kmax "-eps" eps "-bootpct" bootpct "-maxiterations"
> maxiterations "-maxtries" maxtries;
> 
> 
> 
> 
> }
> 
> 
> 
> 
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
> 
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
> 
> 
> 
> 
> foreach f,i in dataFiles {
> 
> 
> 
> 
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
> 
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
> 
> o = processData(f, metaFiles[i], l);
> 
> 
> 
> 
> }
> 
> 
> 
> 
> this configuration causes processData to be invoked as:
> 
> 
> 
> 
> out/clusFile.0.clus = processData(dataFile0.dat, dataFile0.dat,
> out/logFile.0.log);
> 
> 
> 
> 
> if i switch around the oder of the filesys_mappers so that the snippet
> reads:
> 
> 
> 
> 
> metaFile metaFiles[] <filesys_mapper;location="in",sufix="meta">;
> 
> dataFile dataFiles[] <filesys_mapper;location="in",sufix="dat">;
> 
> 
> 
> 
> foreach f,i in dataFiles {
> 
> 
> 
> 
> clusFile o<single_file_mapper; location="out",
> file=@strcat("out/clusFile.",i,".clus")>;
> 
> logFile l<single_file_mapper; location="out",
> file=@strcat("out/logFile.",i,".log")>;
> 
> o = processData(f, metaFiles[i], l);
> 
> 
> 
> 
> }
> 
> 
> 
> 
> the app invocation is called as:
> 
> 
> 
> 
> out/clusFile.o.clus = processData(dataFile0.meta, dataFile0.meta,
> out/logFile.0.log);
> 
> 
> 
> 
> I guess the real question is this: what is the most appropriate way in
> swift to pass into a app invocation two corresponding input files?
> Ideally, it would be something like 'foreach file1,file2,i in
> inputFiles[][] { ? } but that doesn't really make too much sense
> either.
> 
> 
> 
> 
> Anyway, any ideas would be appreciated.
> 
> 
> 
> 
> Cheers,
> 
> Robin
> 
> 
> 
> 
> 
> --
> 
> Robin M. Weiss
> Research Programmer
> Research Computing Center
> The University of Chicago
> 6030 S. Ellis Ave., Suite 289C
> Chicago, IL 60637
> robinweiss at uchicago.edu
> 773.702.9030
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory



From iraicu at cs.iit.edu  Tue Jul 31 16:46:43 2012
From: iraicu at cs.iit.edu (Ioan Raicu)
Date: Tue, 31 Jul 2012 16:46:43 -0500
Subject: [Swift-user] Call for Participation: IEEE eScience 2012 in Chicago,
 IL, October 8-12, 2012
Message-ID: <50185243.2010909@cs.iit.edu>

*Call for Participation*

*IEEE eScience 2012*

*http://www.ci.uchicago.edu/escience2012/index.php*

*October 8^th -12^th , 2012 -- Chicago, IL, USA*

The 8th IEEE International Conference on eScience (eScience 2012) will 
be held at the Hyatt Regency Chicago, Chicago, Illinois, *8-12 October 
2012*, with workshops (including the Microsoft eScience Workshop) on 8-9 
October and the main conference events on 10-12 October.

Scientific research is increasingly carried out by communities of 
researchers that span disciplines, laboratories, organizations and 
national boundaries. These activities involve geographically distributed 
and heterogeneous resources such as computational systems, scientific 
instruments, databases, sensors, software components, networks, and 
people. Such large-scale and enhanced scientific endeavors are carried 
out via collaborations on a global scale in which information and 
computing technology plays a vital role and are thus popularly termed as 
e-Science.


        Keynote Speakers

Description: Professor Gerhard Klimeck


          Professor Gerhard Klimeck

Director of the Network for Computational Nanotechnology and Professor 
of Electrical and Computer Engineering
/Purdue University///






Description: Professor Leonard Smith


          Professor Leonard Smith

Director of the Centre for the Analysis of Time Series (CATS)
/London School of Economics and Political Science/






Description: Dr. Gregory Wilson


          Dr. Gregory Wilson

Software Carpentry
/Mozilla Foundation/





Description: Professor Carole Goble


          Professor Carole Goble

/University of Manchester, UK/





      Workshops

eScience 2012 will include workshops on 8-9 October. In addition to the 
Microsoft eScience Workshop, other workshops were solicited in the Call 
for Workshops 
<http://www.ci.uchicago.edu/escience2012/call_for_workshops.php>. Six 
workshops were accepted:

  * 8 October: Extending High-Performance Computing Beyond its
    Traditional User Communities
    <http://www.psc.edu/index.php/escience-2012-workshop>, Papers due 6
    August, Contact Person/email: Sergiu Sanielevici <sergiu at psc.edu>
  * 8 October: 2nd International Workshop on Analyzing and Improving
    Collaborative eScience with Social Networks (eSoN 12)
    <http://www.ci.uchicago.edu/eson2012/>, Papers due 17 August,
    Contact Person/email: Kyle Chard <kyle at ci.uchicago.edu>
  * 8 October: Advances in eHealth
    <http://www.scalalife.eu/content/advances-ehealth-2012-workshop>,
    Abstracts due 4 July, Papers due 11 July, Contact Person/email:
    Rossen Apostolov <rossen at kth.se>
  * 9 October: Maintainable Software Practices in e-Science
    <http://software.ac.uk/maintainable-software-practice-workshop>,
    Papers due 20 July, Contact Person/email: Neil Chue Hong and
    Jennifer Schopf <softwarepractice2012 at easychair.org>
  * 9 October: eScience Meets the Instrument,
    <https://confluence-vre.its.monash.edu.au/display/escience2012/eScience+Meets+the+Instrument>
    Contact Person/email: Richard Farnsworth <rif at aps.anl.gov>
  * 9 October am: Collaborative research using eScience infrastructure
    and high speed networks
    <http://www.surfnet.nl/en/Hybride_netwerk/SURFlichtpaden/Pages/CollaborativeresearchusingeScienceinfrastructureandhighspeednetworks.aspx>,
    Contact Person/email: Peter Hinrich <peter.hinrich at surfnet.nl>

In addition, there will also be one tutorial:

  * 9 October pm: Big Data Processing: Lessons from Industry and
    Applications in Science
    <http://www.ci.uchicago.edu/escience2012/tutorial.php>, Contact
    Person/email: Roger Barga <barga at microsoft.com>

-- 
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel:    1-847-722-0876
Office: 1-312-567-5704
Email:iraicu at cs.iit.edu
Web:http://www.cs.iit.edu/~iraicu/
Web:http://datasys.cs.iit.edu/
=================================================================
=================================================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120731/324b2585/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 95497 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120731/324b2585/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 92700 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120731/324b2585/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 92056 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120731/324b2585/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 82775 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120731/324b2585/attachment-0003.png>