[Swift-devel] duplicated job submission in swift-0.92?

[Michael Wilde]

OK, what am I missing?

int arr[];

arr[0]=1;
arr[1]=2;

foreach a in arr {
  trace("for", a);
}

login1$ swift zz3.swift
Swift svn swift-r4157 cog-r3056

RunID: 20110331-0134-bfzhkgaa
Progress:
SwiftScript trace: for, 2
SwiftScript trace: for, 1
SwiftScript trace: for, 1
SwiftScript trace: for, 2
Final status:

When did the foreach loop become the twice-each loop?

I need to try some other revision and hosts with this.

- Mike


----- Original Message -----
> Wow, I didn't know we can do that! I treat the docs too canonically :P
> 
> 2011/3/30 Michael Wilde <wilde at mcs.anl.gov>:
> 
> > login1$ cat zz2.swift
> >
> > foreach a in [0:3] {
> >  trace("for", a);
> > }
> >
> > login1$ swift zz2.swift
> > Swift svn swift-r4157 cog-r3056
> >
> > RunID: 20110331-0057-huo8jei0
> > Progress:
> > SwiftScript trace: for, 1
> > SwiftScript trace: for, 3
> > SwiftScript trace: for, 2
> > SwiftScript trace: for, 0
> > Final status:
> > login1$
> >
> > I suspect we need to make this more clear in the user guide and
> > tutorials :)
> 
> I agree.
> 
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> >> Or just use the concurrent mapper to let swift handle the output
> >> naming itself. The resume files can't persist through multiple
> >> sessions though.
> >>
> >> 2011/3/30 Michael Wilde <wilde at mcs.anl.gov>:
> >> > The most common case for this error occurs when two iterations
> >> > within a foreach loop map an output file to the same physical
> >> > file
> >> > name. When swift runs and tries to put the output object into its
> >> > site cache, it sees that a file of the name name is already in
> >> > the
> >> > cache, and its semantics do not allow that.
> >> >
> >> > I have not yet stared at this code long enough to see if this
> >> > explains what is happening here.
> >> >
> >> > I also dont know why it might work under one version and fail
> >> > under
> >> > 0.92. If the above situation is occurring, perhaps there is some
> >> > randomness involved: loop iteration ordering; filename generation
> >> > randomness or difference, etc.
> >> >
> >> > But I would debug with that in mind: make sure that all *output*
> >> > fie
> >> > names mapped by the script are unique. Ideally, one should be
> >> > able
> >> > to find the culprit by grepping the swift log for all the mapped
> >> > file names and look for duplicates.
> >> >
> >> > - Mike
> >> >
> >> >
> >> > ----- Original Message -----
> >> >> Or maybe local variables are static? Maybe they mapped to
> >> >> different
> >> >> files but to the same cache object? But I have been doing local
> >> >> variables in my own workflows though.
> >> >>
> >> >> 2011/3/30 Jonathan Monette <jon.monette at gmail.com>:
> >> >> > Ok. I understand this error better. But shouldn't that be a
> >> >> > different
> >> >> > error then? Like a and b are mapped to the same file? I don't
> >> >> > know
> >> >> > if Swift
> >> >> > can know this but looking at the explanation and error it
> >> >> > should
> >> >> > unless this
> >> >> > cache message has a deeper meaning.
> >> >> >
> >> >> > On Wed, Mar 30, 2011 at 6:21 PM, Allan Espinosa
> >> >> > <aespinosa at cs.uchicago.edu>
> >> >> > wrote:
> >> >> >>
> >> >> >> I had this error before when two output mapper objects mapped
> >> >> >> to
> >> >> >> the same
> >> >> >> file.
> >> >> >>
> >> >> >> $ swift bug_same.swift
> >> >> >> Swift svn swift-r4208 cog-r3073
> >> >> >>
> >> >> >> RunID: 20110330-1818-ygec7ppa
> >> >> >> Progress: time:0
> >> >> >> The cache already contains
> >> >> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
> >> >> >>
> >> >> >> The cache already contains
> >> >> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
> >> >> >>
> >> >> >> Progress: time:1960 Stage in:1 Finished successfully:1
> >> >> >> The cache already contains
> >> >> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
> >> >> >>
> >> >> >> [aespinosa at communicado testing]$
> >> >> >> [aespinosa at communicado testing]$ cat bug_same.swift
> >> >> >> type file;
> >> >> >>
> >> >> >> app (file out) echo(string input) {
> >> >> >>  echo input stdout=@filename(out);
> >> >> >> }
> >> >> >>
> >> >> >> file a <"foo">;
> >> >> >> file b <"foo">;
> >> >> >>
> >> >> >> a = echo("hello world");
> >> >> >> b = echo("foo bar");
> >> >> >>
> >> >> >> But i think you should be using other Swift mappers that does
> >> >> >> auto-numbering of files by default.
> >> >> >>
> >> >> >> -Allan
> >> >> >>
> >> >> >> 2011/3/30 Zhao Zhang <zhaozhang at uchicago.edu>:
> >> >> >> > Hi guys,
> >> >> >> >
> >> >> >> > I am seeing something weird in swfit-0.92. Any idea about
> >> >> >> > this?
> >> >> >> > The swift script is very simple:
> >> >> >> >
> >> >> >> > zzhang at sandbox:~/workplace/Andrey> cat movies.swift
> >> >> >> > type Pickle {}
> >> >> >> > type History {}
> >> >> >> > type Image {}
> >> >> >> >
> >> >> >> > app (History historyout) movie_graph (int rerun, int
> >> >> >> > epochs,
> >> >> >> > Pickle
> >> >> >> > picklefile)
> >> >> >> > {
> >> >> >> >   movie_graph rerun epochs;
> >> >> >> > }
> >> >> >> >
> >> >> >> > int arr[];
> >> >> >> > iterate i
> >> >> >> > {
> >> >> >> >  arr[i] = i+1;
> >> >> >> > }until(i == 1);
> >> >> >> >
> >> >> >> > int epochs;
> >> >> >> > epochs = 3;
> >> >> >> > Pickle picklefile <single_file_mapper;
> >> >> >> > file="for_movies.pickled">;
> >> >> >> > foreach a in arr{
> >> >> >> >  History historyout <single_file_mapper;
> >> >> >> >  file=@strcat("output/rerun", a,
> >> >> >> > "/histories.pickled-", a)>;
> >> >> >> >  historyout = movie_graph(a, epochs, picklefile);
> >> >> >> > }
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > I ran the script with the latest 0.92 version, which is
> >> >> >> > loaded
> >> >> >> > as
> >> >> >> > a
> >> >> >> > module
> >> >> >> > on beagle. The I saw this:
> >> >> >> > zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
> >> >> >> > movies.swift
> >> >> >> > Variable epochs defined in scope 99878388 shadows variable
> >> >> >> > of
> >> >> >> > same name
> >> >> >> > in
> >> >> >> > scope 1813605401
> >> >> >> > Variable picklefile defined in scope 99878388 shadows
> >> >> >> > variable
> >> >> >> > of
> >> >> >> > same
> >> >> >> > name
> >> >> >> > in scope 1813605401
> >> >> >> > Swift svn swift-r4157 cog-r3056
> >> >> >> >
> >> >> >> > RunID: 20110330-1636-ev8vm8gb
> >> >> >> > Progress:
> >> >> >> > Progress: Selecting site:3 Active:1
> >> >> >> > Progress: Selecting site:3 Checking status:1
> >> >> >> > Progress: Selecting site:2 Stage in:1 Finished
> >> >> >> > successfully:1
> >> >> >> > Progress: Selecting site:2 Active:1 Finished successfully:1
> >> >> >> > Progress: Selecting site:2 Active:1 Finished successfully:1
> >> >> >> > Progress: Selecting site:1 Stage in:1 Finished
> >> >> >> > successfully:2
> >> >> >> > Progress: Selecting site:1 Active:1 Finished successfully:2
> >> >> >> > Progress: Selecting site:1 Checking status:1 Finished
> >> >> >> > successfully:2
> >> >> >> > The cache already contains
> >> >> >> >
> >> >> >> > localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
> >> >> >> >
> >> >> >> > Execution failed:
> >> >> >> >        The cache already contains
> >> >> >> >
> >> >> >> > localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
> >> >> >> >
> >> >> >> >
> >> >> >> > Then I switched to an older version, it worked well.
> >> >> >> > zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
> >> >> >> > movies.swift
> >> >> >> > Variable epochs defined in scope 212602028 shadows variable
> >> >> >> > of
> >> >> >> > same name
> >> >> >> > in
> >> >> >> > scope 1538939834
> >> >> >> > Variable picklefile defined in scope 212602028 shadows
> >> >> >> > variable
> >> >> >> > of same
> >> >> >> > name
> >> >> >> > in scope 1538939834
> >> >> >> > Swift svn swift-r3291 (swift modified locally) cog-r2750
> >> >> >> > (cog
> >> >> >> > modified
> >> >> >> > locally)
> >> >> >> >
> >> >> >> > RunID: 20110330-1639-gmbyz1qa
> >> >> >> > Progress:
> >> >> >> > Progress: Active:2
> >> >> >> > Progress: Active:1 Checking status:1
> >> >> >> > Final status: Finished successfully:2
> >> >> >> _______________________________________________
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory