[Swift-devel] duplicated job submission in swift-0.92?

[Zhao Zhang]

Yeah, foreach a in [1:3] works perfectly.

best
zhao

On 3/30/2011 7:58 PM, Michael Wilde wrote:
> I wouldn't do that quite yet. This is very curious, but I think its the likely cause of what you are seeing.  The foreach() loop in this case seems to be having double vision :)  Its either some Swift subtlety or a bug:
>
> login1$ cat zz1.swift
> int arr[];
>
> iterate i
> {
>     arr[i] = i+1;
>     trace(i, arr[i]);
> }until(i == 1);
>
> foreach a in arr {
>    trace("for", a);
> }
>
> login1$ swift zz1.swift
> Swift svn swift-r4157 cog-r3056
>
> RunID: 20110331-0053-tkh8yla5
> Progress:
> SwiftScript trace: 0, 1
> SwiftScript trace: for, 1
> SwiftScript trace: for, 1
> SwiftScript trace: 1, 2
> SwiftScript trace: for, 2
> SwiftScript trace: for, 2
> Final status:
> login1$
>
>
> But, Zhao, you could in the meantime use much simpler code like so:
>
> login1$ cat zz2.swift
>
> foreach a in [0:3] {
>    trace("for", a);
> }
>
> login1$ swift zz2.swift
> Swift svn swift-r4157 cog-r3056
>
> RunID: 20110331-0057-huo8jei0
> Progress:
> SwiftScript trace: for, 1
> SwiftScript trace: for, 3
> SwiftScript trace: for, 2
> SwiftScript trace: for, 0
> Final status:
> login1$
>
> I suspect we need to make this more clear in the user guide and tutorials :)
>
> - Mike
>
>
> ----- Original Message -----
>> Or just use the concurrent mapper to let swift handle the output
>> naming itself. The resume files can't persist through multiple
>> sessions though.
>>
>> 2011/3/30 Michael Wilde<wilde at mcs.anl.gov>:
>>> The most common case for this error occurs when two iterations
>>> within a foreach loop map an output file to the same physical file
>>> name. When swift runs and tries to put the output object into its
>>> site cache, it sees that a file of the name name is already in the
>>> cache, and its semantics do not allow that.
>>>
>>> I have not yet stared at this code long enough to see if this
>>> explains what is happening here.
>>>
>>> I also dont know why it might work under one version and fail under
>>> 0.92. If the above situation is occurring, perhaps there is some
>>> randomness involved: loop iteration ordering; filename generation
>>> randomness or difference, etc.
>>>
>>> But I would debug with that in mind: make sure that all *output* fie
>>> names mapped by the script are unique. Ideally, one should be able
>>> to find the culprit by grepping the swift log for all the mapped
>>> file names and look for duplicates.
>>>
>>> - Mike
>>>
>>>
>>> ----- Original Message -----
>>>> Or maybe local variables are static? Maybe they mapped to different
>>>> files but to the same cache object? But I have been doing local
>>>> variables in my own workflows though.
>>>>
>>>> 2011/3/30 Jonathan Monette<jon.monette at gmail.com>:
>>>>> Ok. I understand this error better. But shouldn't that be a
>>>>> different
>>>>> error then? Like a and b are mapped to the same file? I don't
>>>>> know
>>>>> if Swift
>>>>> can know this but looking at the explanation and error it should
>>>>> unless this
>>>>> cache message has a deeper meaning.
>>>>>
>>>>> On Wed, Mar 30, 2011 at 6:21 PM, Allan Espinosa
>>>>> <aespinosa at cs.uchicago.edu>
>>>>> wrote:
>>>>>> I had this error before when two output mapper objects mapped to
>>>>>> the same
>>>>>> file.
>>>>>>
>>>>>> $ swift bug_same.swift
>>>>>> Swift svn swift-r4208 cog-r3073
>>>>>>
>>>>>> RunID: 20110330-1818-ygec7ppa
>>>>>> Progress: time:0
>>>>>> The cache already contains
>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>
>>>>>> The cache already contains
>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>
>>>>>> Progress: time:1960 Stage in:1 Finished successfully:1
>>>>>> The cache already contains
>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>
>>>>>> [aespinosa at communicado testing]$
>>>>>> [aespinosa at communicado testing]$ cat bug_same.swift
>>>>>> type file;
>>>>>>
>>>>>> app (file out) echo(string input) {
>>>>>>   echo input stdout=@filename(out);
>>>>>> }
>>>>>>
>>>>>> file a<"foo">;
>>>>>> file b<"foo">;
>>>>>>
>>>>>> a = echo("hello world");
>>>>>> b = echo("foo bar");
>>>>>>
>>>>>> But i think you should be using other Swift mappers that does
>>>>>> auto-numbering of files by default.
>>>>>>
>>>>>> -Allan
>>>>>>
>>>>>> 2011/3/30 Zhao Zhang<zhaozhang at uchicago.edu>:
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> I am seeing something weird in swfit-0.92. Any idea about
>>>>>>> this?
>>>>>>> The swift script is very simple:
>>>>>>>
>>>>>>> zzhang at sandbox:~/workplace/Andrey>  cat movies.swift
>>>>>>> type Pickle {}
>>>>>>> type History {}
>>>>>>> type Image {}
>>>>>>>
>>>>>>> app (History historyout) movie_graph (int rerun, int epochs,
>>>>>>> Pickle
>>>>>>> picklefile)
>>>>>>> {
>>>>>>>    movie_graph rerun epochs;
>>>>>>> }
>>>>>>>
>>>>>>> int arr[];
>>>>>>> iterate i
>>>>>>> {
>>>>>>>   arr[i] = i+1;
>>>>>>> }until(i == 1);
>>>>>>>
>>>>>>> int epochs;
>>>>>>> epochs = 3;
>>>>>>> Pickle picklefile<single_file_mapper;
>>>>>>> file="for_movies.pickled">;
>>>>>>> foreach a in arr{
>>>>>>>   History historyout<single_file_mapper;
>>>>>>>   file=@strcat("output/rerun", a,
>>>>>>> "/histories.pickled-", a)>;
>>>>>>>   historyout = movie_graph(a, epochs, picklefile);
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I ran the script with the latest 0.92 version, which is loaded
>>>>>>> as
>>>>>>> a
>>>>>>> module
>>>>>>> on beagle. The I saw this:
>>>>>>> zzhang at sandbox:~/workplace/Andrey>  swift -tc.file ./tc.data
>>>>>>> movies.swift
>>>>>>> Variable epochs defined in scope 99878388 shadows variable of
>>>>>>> same name
>>>>>>> in
>>>>>>> scope 1813605401
>>>>>>> Variable picklefile defined in scope 99878388 shadows variable
>>>>>>> of
>>>>>>> same
>>>>>>> name
>>>>>>> in scope 1813605401
>>>>>>> Swift svn swift-r4157 cog-r3056
>>>>>>>
>>>>>>> RunID: 20110330-1636-ev8vm8gb
>>>>>>> Progress:
>>>>>>> Progress: Selecting site:3 Active:1
>>>>>>> Progress: Selecting site:3 Checking status:1
>>>>>>> Progress: Selecting site:2 Stage in:1 Finished successfully:1
>>>>>>> Progress: Selecting site:2 Active:1 Finished successfully:1
>>>>>>> Progress: Selecting site:2 Active:1 Finished successfully:1
>>>>>>> Progress: Selecting site:1 Stage in:1 Finished successfully:2
>>>>>>> Progress: Selecting site:1 Active:1 Finished successfully:2
>>>>>>> Progress: Selecting site:1 Checking status:1 Finished
>>>>>>> successfully:2
>>>>>>> The cache already contains
>>>>>>>
>>>>>>> localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>>>>>>>
>>>>>>> Execution failed:
>>>>>>>         The cache already contains
>>>>>>>
>>>>>>> localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>>>>>>>
>>>>>>>
>>>>>>> Then I switched to an older version, it worked well.
>>>>>>> zzhang at sandbox:~/workplace/Andrey>  swift -tc.file ./tc.data
>>>>>>> movies.swift
>>>>>>> Variable epochs defined in scope 212602028 shadows variable of
>>>>>>> same name
>>>>>>> in
>>>>>>> scope 1538939834
>>>>>>> Variable picklefile defined in scope 212602028 shadows
>>>>>>> variable
>>>>>>> of same
>>>>>>> name
>>>>>>> in scope 1538939834
>>>>>>> Swift svn swift-r3291 (swift modified locally) cog-r2750 (cog
>>>>>>> modified
>>>>>>> locally)
>>>>>>>
>>>>>>> RunID: 20110330-1639-gmbyz1qa
>>>>>>> Progress:
>>>>>>> Progress: Active:2
>>>>>>> Progress: Active:1 Checking status:1
>>>>>>> Final status: Finished successfully:2
>>>>>> _______________________________________________