[Swift-user] moving files during runtime

[Michael Wilde]

Khushbu, this example works for me:

$ cat serial.swift

type file;

app (file o) incfile (file i)
{
  sh "-c" "echo $(($(cat) + 1))" stdin=@i stdout=@o;
}
  
file f[];
file first<"f.0">;

f[0] = first;
foreach i in [1:10] {
  file newf<single_file_mapper; file=@strcat("f.",i)>;
  newf = incfile(f[i-1]);
  f[i] = newf;
}

$ cat tsh
localhost sh /bin/sh

$ swift -tc.file tsh serial.swift
no sites file specified, setting to default: /software/swift-0.93-all/etc/sites.xml
Swift 0.93 swift-r5483 cog-r3339

RunID: 20120711-0259-mkjdrq1g
Progress:  time: Wed, 11 Jul 2012 03:00:00 +0000
Final status:  time: Wed, 11 Jul 2012 03:00:01 +0000  Finished successfully:10

$ more f.*
::::::::::::::
f.0
::::::::::::::
100
::::::::::::::
f.1
::::::::::::::
101
::::::::::::::
f.10
::::::::::::::
110
::::::::::::::
f.2
::::::::::::::
102
::::::::::::::
f.3
::::::::::::::
103
::::::::::::::
f.4
::::::::::::::
104
::::::::::::::
f.5
::::::::::::::
105
::::::::::::::
f.6
::::::::::::::
106
::::::::::::::
f.7
::::::::::::::
107
::::::::::::::
f.8
::::::::::::::
108
::::::::::::::
f.9
::::::::::::::
109
$ 


----- Original Message -----
> From: "Michael Wilde" <wilde at mcs.anl.gov>
> To: "Khushbu Agarwal" <Khushbu.Agarwal at pnnl.gov>
> Cc: "Karen L Schuchardt" <Karen.Schuchardt at pnnl.gov>, swift-user at ci.uchicago.edu
> Sent: Tuesday, July 10, 2012 9:44:43 PM
> Subject: Re: [Swift-user] moving files during runtime
> Hi Khushbu,
> 
> > ...In the shell script for the app, I actually do a
> > move of the file that was produced in my previous iteration to my
> > current iteration. However when Swift runs, I just see lot of links.
> > Basically file in iteration 4 has a symbolic link to file in
> > iteration
> > 3, 3 to 2 and so on. This works ok for couple of iterations, but
> > after
> > a while system seem to fail with message “ unable to follow links”.
> > Since we are trying to run over 100’s of iterations, I would like to
> > know if there is a way to tell swift to just copy or move the file
> > than to create symbolic links.
> 
> I'm assuming that since each iteration of this process depends on the
> prior iteration, they all run serially.
> 
> If thats the case, you can use either a foreach loop or an iterate
> statement, and just let Swift do all the copying in a natural way:
> 
> type file;
> file f[];
> file first<"initial.data">;
> 
> f[0] = first;
> foreach i in [1:100] {
> f[i] = myApp(f[i-1]);
> }
> 
> (I didnt test this, but something similar should work for your
> application)
> 
> Whats happening in your current case is that your app script is
> "breaking the rules" by trying to preserve data objects from the Swift
> pre-job "sandbox" directory (as explained in User Guide sections 5.1
> and 5.2:
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_mapping_of_app_semantics_into_unix_process_execution_semantics
> 
> and
> 
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_how_swift_implements_the_site_execution_model
> 
> I could explain a few ways to make your current approach work, but I
> think the approach above is much more natural and also safer and less
> brittle - less vulnerable to breakage if we change the Swift runtime
> model. (For example, you could do the cp in your app script rather
> than an mv, or you could Swift to do cp's rather than links by
> specifying the "scratch" tag in your sites.xml entry. But I didn't
> tell you either of these things because they are not good practice ;)
> 
> - Mike
> 
> 
> 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory