[Swift-devel] duplicated job submission in swift-0.92?
Justin M Wozniak
wozniak at mcs.anl.gov
Thu Mar 31 09:34:14 CDT 2011
I did that to get things unified in trunk. I can revert things back to
the state before that if you think it would help sort this out.
On Wed, 30 Mar 2011, Mihael Hategan wrote:
> Why was trunk merged into the stable branch?
>
> On Wed, 2011-03-30 at 18:40 -0700, Mihael Hategan wrote:
>> On Wed, 2011-03-30 at 20:37 -0500, Michael Wilde wrote:
>>> OK, what am I missing?
>>
>> Nothing. That shouldn't be happening.
>>
>> Here's what (approximately) you should get:
>> Swift svn swift-r3526 (swift modified locally) cog-r656 (cog modified
>> locally)
>>
>> RunID: 20110330-1839-y71gls4a
>> Progress: time:0
>> [Misc] WARN pool-1-thread-4 - SwiftScript trace: for, 2
>> [Misc] WARN pool-1-thread-1 - SwiftScript trace: for, 1
>> Final status: time:54
>>
>>>
>>> int arr[];
>>>
>>> arr[0]=1;
>>> arr[1]=2;
>>>
>>> foreach a in arr {
>>> trace("for", a);
>>> }
>>>
>>> login1$ swift zz3.swift
>>> Swift svn swift-r4157 cog-r3056
>>>
>>> RunID: 20110331-0134-bfzhkgaa
>>> Progress:
>>> SwiftScript trace: for, 2
>>> SwiftScript trace: for, 1
>>> SwiftScript trace: for, 1
>>> SwiftScript trace: for, 2
>>> Final status:
>>>
>>> When did the foreach loop become the twice-each loop?
>>>
>>> I need to try some other revision and hosts with this.
>>>
>>> - Mike
>>>
>>>
>>> ----- Original Message -----
>>>> Wow, I didn't know we can do that! I treat the docs too canonically :P
>>>>
>>>> 2011/3/30 Michael Wilde <wilde at mcs.anl.gov>:
>>>>
>>>>> login1$ cat zz2.swift
>>>>>
>>>>> foreach a in [0:3] {
>>>>> trace("for", a);
>>>>> }
>>>>>
>>>>> login1$ swift zz2.swift
>>>>> Swift svn swift-r4157 cog-r3056
>>>>>
>>>>> RunID: 20110331-0057-huo8jei0
>>>>> Progress:
>>>>> SwiftScript trace: for, 1
>>>>> SwiftScript trace: for, 3
>>>>> SwiftScript trace: for, 2
>>>>> SwiftScript trace: for, 0
>>>>> Final status:
>>>>> login1$
>>>>>
>>>>> I suspect we need to make this more clear in the user guide and
>>>>> tutorials :)
>>>>
>>>> I agree.
>>>>
>>>>>
>>>>> - Mike
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> Or just use the concurrent mapper to let swift handle the output
>>>>>> naming itself. The resume files can't persist through multiple
>>>>>> sessions though.
>>>>>>
>>>>>> 2011/3/30 Michael Wilde <wilde at mcs.anl.gov>:
>>>>>>> The most common case for this error occurs when two iterations
>>>>>>> within a foreach loop map an output file to the same physical
>>>>>>> file
>>>>>>> name. When swift runs and tries to put the output object into its
>>>>>>> site cache, it sees that a file of the name name is already in
>>>>>>> the
>>>>>>> cache, and its semantics do not allow that.
>>>>>>>
>>>>>>> I have not yet stared at this code long enough to see if this
>>>>>>> explains what is happening here.
>>>>>>>
>>>>>>> I also dont know why it might work under one version and fail
>>>>>>> under
>>>>>>> 0.92. If the above situation is occurring, perhaps there is some
>>>>>>> randomness involved: loop iteration ordering; filename generation
>>>>>>> randomness or difference, etc.
>>>>>>>
>>>>>>> But I would debug with that in mind: make sure that all *output*
>>>>>>> fie
>>>>>>> names mapped by the script are unique. Ideally, one should be
>>>>>>> able
>>>>>>> to find the culprit by grepping the swift log for all the mapped
>>>>>>> file names and look for duplicates.
>>>>>>>
>>>>>>> - Mike
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> Or maybe local variables are static? Maybe they mapped to
>>>>>>>> different
>>>>>>>> files but to the same cache object? But I have been doing local
>>>>>>>> variables in my own workflows though.
>>>>>>>>
>>>>>>>> 2011/3/30 Jonathan Monette <jon.monette at gmail.com>:
>>>>>>>>> Ok. I understand this error better. But shouldn't that be a
>>>>>>>>> different
>>>>>>>>> error then? Like a and b are mapped to the same file? I don't
>>>>>>>>> know
>>>>>>>>> if Swift
>>>>>>>>> can know this but looking at the explanation and error it
>>>>>>>>> should
>>>>>>>>> unless this
>>>>>>>>> cache message has a deeper meaning.
>>>>>>>>>
>>>>>>>>> On Wed, Mar 30, 2011 at 6:21 PM, Allan Espinosa
>>>>>>>>> <aespinosa at cs.uchicago.edu>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I had this error before when two output mapper objects mapped
>>>>>>>>>> to
>>>>>>>>>> the same
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> $ swift bug_same.swift
>>>>>>>>>> Swift svn swift-r4208 cog-r3073
>>>>>>>>>>
>>>>>>>>>> RunID: 20110330-1818-ygec7ppa
>>>>>>>>>> Progress: time:0
>>>>>>>>>> The cache already contains
>>>>>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>>>>>
>>>>>>>>>> The cache already contains
>>>>>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>>>>>
>>>>>>>>>> Progress: time:1960 Stage in:1 Finished successfully:1
>>>>>>>>>> The cache already contains
>>>>>>>>>> localhost:bug_same-20110330-1818-ygec7ppa/shared/foo.
>>>>>>>>>>
>>>>>>>>>> [aespinosa at communicado testing]$
>>>>>>>>>> [aespinosa at communicado testing]$ cat bug_same.swift
>>>>>>>>>> type file;
>>>>>>>>>>
>>>>>>>>>> app (file out) echo(string input) {
>>>>>>>>>> echo input stdout=@filename(out);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> file a <"foo">;
>>>>>>>>>> file b <"foo">;
>>>>>>>>>>
>>>>>>>>>> a = echo("hello world");
>>>>>>>>>> b = echo("foo bar");
>>>>>>>>>>
>>>>>>>>>> But i think you should be using other Swift mappers that does
>>>>>>>>>> auto-numbering of files by default.
>>>>>>>>>>
>>>>>>>>>> -Allan
>>>>>>>>>>
>>>>>>>>>> 2011/3/30 Zhao Zhang <zhaozhang at uchicago.edu>:
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> I am seeing something weird in swfit-0.92. Any idea about
>>>>>>>>>>> this?
>>>>>>>>>>> The swift script is very simple:
>>>>>>>>>>>
>>>>>>>>>>> zzhang at sandbox:~/workplace/Andrey> cat movies.swift
>>>>>>>>>>> type Pickle {}
>>>>>>>>>>> type History {}
>>>>>>>>>>> type Image {}
>>>>>>>>>>>
>>>>>>>>>>> app (History historyout) movie_graph (int rerun, int
>>>>>>>>>>> epochs,
>>>>>>>>>>> Pickle
>>>>>>>>>>> picklefile)
>>>>>>>>>>> {
>>>>>>>>>>> movie_graph rerun epochs;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> int arr[];
>>>>>>>>>>> iterate i
>>>>>>>>>>> {
>>>>>>>>>>> arr[i] = i+1;
>>>>>>>>>>> }until(i == 1);
>>>>>>>>>>>
>>>>>>>>>>> int epochs;
>>>>>>>>>>> epochs = 3;
>>>>>>>>>>> Pickle picklefile <single_file_mapper;
>>>>>>>>>>> file="for_movies.pickled">;
>>>>>>>>>>> foreach a in arr{
>>>>>>>>>>> History historyout <single_file_mapper;
>>>>>>>>>>> file=@strcat("output/rerun", a,
>>>>>>>>>>> "/histories.pickled-", a)>;
>>>>>>>>>>> historyout = movie_graph(a, epochs, picklefile);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I ran the script with the latest 0.92 version, which is
>>>>>>>>>>> loaded
>>>>>>>>>>> as
>>>>>>>>>>> a
>>>>>>>>>>> module
>>>>>>>>>>> on beagle. The I saw this:
>>>>>>>>>>> zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
>>>>>>>>>>> movies.swift
>>>>>>>>>>> Variable epochs defined in scope 99878388 shadows variable
>>>>>>>>>>> of
>>>>>>>>>>> same name
>>>>>>>>>>> in
>>>>>>>>>>> scope 1813605401
>>>>>>>>>>> Variable picklefile defined in scope 99878388 shadows
>>>>>>>>>>> variable
>>>>>>>>>>> of
>>>>>>>>>>> same
>>>>>>>>>>> name
>>>>>>>>>>> in scope 1813605401
>>>>>>>>>>> Swift svn swift-r4157 cog-r3056
>>>>>>>>>>>
>>>>>>>>>>> RunID: 20110330-1636-ev8vm8gb
>>>>>>>>>>> Progress:
>>>>>>>>>>> Progress: Selecting site:3 Active:1
>>>>>>>>>>> Progress: Selecting site:3 Checking status:1
>>>>>>>>>>> Progress: Selecting site:2 Stage in:1 Finished
>>>>>>>>>>> successfully:1
>>>>>>>>>>> Progress: Selecting site:2 Active:1 Finished successfully:1
>>>>>>>>>>> Progress: Selecting site:2 Active:1 Finished successfully:1
>>>>>>>>>>> Progress: Selecting site:1 Stage in:1 Finished
>>>>>>>>>>> successfully:2
>>>>>>>>>>> Progress: Selecting site:1 Active:1 Finished successfully:2
>>>>>>>>>>> Progress: Selecting site:1 Checking status:1 Finished
>>>>>>>>>>> successfully:2
>>>>>>>>>>> The cache already contains
>>>>>>>>>>>
>>>>>>>>>>> localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>>>>>>>>>>>
>>>>>>>>>>> Execution failed:
>>>>>>>>>>> The cache already contains
>>>>>>>>>>>
>>>>>>>>>>> localhost:movies-20110330-1636-ev8vm8gb/shared/output/rerun1/histories.pickled-1.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then I switched to an older version, it worked well.
>>>>>>>>>>> zzhang at sandbox:~/workplace/Andrey> swift -tc.file ./tc.data
>>>>>>>>>>> movies.swift
>>>>>>>>>>> Variable epochs defined in scope 212602028 shadows variable
>>>>>>>>>>> of
>>>>>>>>>>> same name
>>>>>>>>>>> in
>>>>>>>>>>> scope 1538939834
>>>>>>>>>>> Variable picklefile defined in scope 212602028 shadows
>>>>>>>>>>> variable
>>>>>>>>>>> of same
>>>>>>>>>>> name
>>>>>>>>>>> in scope 1538939834
>>>>>>>>>>> Swift svn swift-r3291 (swift modified locally) cog-r2750
>>>>>>>>>>> (cog
>>>>>>>>>>> modified
>>>>>>>>>>> locally)
>>>>>>>>>>>
>>>>>>>>>>> RunID: 20110330-1639-gmbyz1qa
>>>>>>>>>>> Progress:
>>>>>>>>>>> Progress: Active:2
>>>>>>>>>>> Progress: Active:1 Checking status:1
>>>>>>>>>>> Final status: Finished successfully:2
>>>>>>>>>> _______________________________________________
>>>>>
>>>
>>
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>
--
Justin M Wozniak
More information about the Swift-devel
mailing list