[Swift-devel] duplicated job submission in swift-0.92?

[Allan Espinosa]

Or just use the concurrent mapper to let swift handle the output
naming itself.  The resume files can't persist through multiple
sessions though.

2011/3/30 Michael Wilde <wilde at mcs.anl.gov>:
> The most common case for this error occurs when two iterations within a foreach loop map an output file to the same physical file name. When swift runs and tries to put the output object into its site cache, it sees that a file of the name name is already in the cache, and its semantics do not allow that.
>
> I have not yet stared at this code long enough to see if this explains what is happening here.
>
> I also dont know why it might work under one version and fail under 0.92. If the above situation is occurring, perhaps there is some randomness involved: loop iteration ordering; filename generation randomness or difference, etc.
>
> But I would debug with that in mind: make sure that all *output* fie names mapped by the script are unique.  Ideally, one should be able to find the culprit by grepping the swift log for all the mapped file names and look for duplicates.
>
> - Mike
>
>
> ----- Original Message -----
>> Or maybe local variables are static? Maybe they mapped to different
>> files but to the same cache object? But I have been doing local
>> variables in my own workflows though.
>>
>> 2011/3/30 Jonathan Monette <jon.monette at gmail.com>:
>> > Ok. I understand this error better. But shouldn't that be a
>> > different
>> > error then? Like a and b are mapped to the same file? I don't know
>> > if Swift
>> > can know this but looking at the explanation and error it should
>> > unless this
>> > cache message has a deeper meaning.
>> >
>> > On Wed, Mar 30, 2011 at 6:21 PM, Allan Espinosa
>> > <aespinosa at cs.uchicago.edu>
>> > wrote:
>> >>
>> >> I had this error before when two output mapper objects mapped to
>> >> the same
>> >> file.
>> >>
>> >> $ swift bug_same.swift
>> >> Swift svn swift-r4208 cog-r3073
>> >>
>> >> RunID: 20110330-1818-ygec7ppa
>> >> Progress: time:0
>> >> The cache already contains
>> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>> >>
>> >> The cache already contains
>> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>> >>
>> >> Progress: time:1960 Stage in:1 Finished successfully:1
>> >> The cache already contains
>> >> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>> >>
>> >> [aespinosa at communicado testing]$
>> >> [aespinosa at communicado testing]$ cat bug_same.swift
>> >> type file;
>> >>
>> >> app (file out) echo(string input) {
>> >>  echo input stdout=@filename(out);
>> >> }
>> >>
>> >> file a <"foo">;
>> >> file b <"foo">;
>> >>
>> >> a = echo("hello world");
>> >> b = echo("foo bar");
>> >>
>> >> But i think you should be using other Swift mappers that does
>> >> auto-numbering of files by default.
>> >>
>> >> -Allan
>> >>
>> >> 2011/3/30 Zhao Zhang <zhaozhang at uchicago.edu>:
>> >> > Hi guys,
>> >> >
>> >> > I am seeing something weird in swfit-0.92. Any idea about this?
>> >> > The swift script is very simple:
>> >> >
>> >> > zzhang at sandbox:~/workplace/Andrey> cat movies.swift
>> >> > type Pickle {}
>> >> > type History {}
>> >> > type Image {}
>> >> >
>> >> > app (History historyout) movie_graph (int rerun, int epochs,
>> >> > Pickle
>> >> > picklefile)
>> >> > {
>> >> >   movie_graph rerun epochs;
>> >> > }
>> >> >
>> >> > int arr[];
>> >> > iterate i
>> >> > {
>> >> >  arr[i] = i+1;
>> >> > }until(i == 1);
>> >> >
>> >> > int epochs;
>> >> > epochs = 3;
>> >> > Pickle picklefile <single_file_mapper;
>> >> > file="for_movies.pickled">;
>> >> > foreach a in arr{
>> >> >  History historyout <single_file_mapper;
>> >> >  file=@strcat("output/rerun", a,
>> >> > "/histories.pickled-", a)>;
>> >> >  historyout = movie_graph(a, epochs, picklefile);
>> >> > }
>> >> >
>> >> >
>> >> >
>> >> > I ran the script with the latest 0.92 version, which is loaded as
>> >> > a
>> >> > module
>> >> > on beagle. The I saw this:
>> >> > zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
>> >> > movies.swift
>> >> > Variable epochs defined in scope 99878388 shadows variable of
>> >> > same name
>> >> > in
>> >> > scope 1813605401
>> >> > Variable picklefile defined in scope 99878388 shadows variable of
>> >> > same
>> >> > name
>> >> > in scope 1813605401
>> >> > Swift svn swift-r4157 cog-r3056
>> >> >
>> >> > RunID: 20110330-1636-ev8vm8gb
>> >> > Progress:
>> >> > Progress: Selecting site:3 Active:1
>> >> > Progress: Selecting site:3 Checking status:1
>> >> > Progress: Selecting site:2 Stage in:1 Finished successfully:1
>> >> > Progress: Selecting site:2 Active:1 Finished successfully:1
>> >> > Progress: Selecting site:2 Active:1 Finished successfully:1
>> >> > Progress: Selecting site:1 Stage in:1 Finished successfully:2
>> >> > Progress: Selecting site:1 Active:1 Finished successfully:2
>> >> > Progress: Selecting site:1 Checking status:1 Finished
>> >> > successfully:2
>> >> > The cache already contains
>> >> >
>> >> > localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>> >> >
>> >> > Execution failed:
>> >> >        The cache already contains
>> >> >
>> >> > localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>> >> >
>> >> >
>> >> > Then I switched to an older version, it worked well.
>> >> > zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
>> >> > movies.swift
>> >> > Variable epochs defined in scope 212602028 shadows variable of
>> >> > same name
>> >> > in
>> >> > scope 1538939834
>> >> > Variable picklefile defined in scope 212602028 shadows variable
>> >> > of same
>> >> > name
>> >> > in scope 1538939834
>> >> > Swift svn swift-r3291 (swift modified locally) cog-r2750 (cog
>> >> > modified
>> >> > locally)
>> >> >
>> >> > RunID: 20110330-1639-gmbyz1qa
>> >> > Progress:
>> >> > Progress: Active:2
>> >> > Progress: Active:1 Checking status:1
>> >> > Final status: Finished successfully:2
>> >> _______________________________________________