[Swift-user] Known issue with stdout and stderr file delays?
Jonathan Ozik
jozik at uchicago.edu
Thu May 29 09:51:46 CDT 2014
Thank you Mihael. These fixes will be very helpful.
Jonathan
On May 29, 2014, at 9:46 AM, Michael Wilde <wilde at anl.gov> wrote:
> Thanks, Mihael. My tests for the two bugs related to app array return mapping now work, on trunk.
>
> 1) returning scalar files along with array returns from the same app
> 2) returning output arrays mapped with simple_mapper without location= argument
>
> Yadu, these tests are on midway in ~wilde/swift/lab/arrmapbug/*swift
>
> - Mike
>
>
> On 5/29/14, 4:26 AM, Mihael Hategan wrote:
>> Thanks. This was only an issue without provider staging.
>>
>> Anyway, the problem was that the dynamic data collect code was
>> overwriting a variable used to point to the directory where the app
>> output files were. Silly shell scoping rules (or the lack thereof), and
>> silly me for not seeing it.
>>
>> Fix committed to trunk.
>>
>> Mihael
>>
>> On Wed, 2014-05-28 at 23:50 -0500, Michael Wilde wrote:
>>> Here's an example of the problem (what looks to me like a Swift bug in
>>> the new array return feature).
>>>
>>> amap01.swift tries to return a file array and a single file (stdout).
>>> Swift is unable to find the mapped stdout file, even though it exists.
>>>
>>> amap02.swift returns just the stdout file (to show that this part of the
>>> test script works).
>>>
>>> amap03.swift returns just the array. It, too, works.
>>>
>>> But returning both together fails, suggesting that the swiftwrap code
>>> that collects the array prevents any other non-array files from being
>>> returned.
>>>
>>> Tests of Jonathan'sscript indicated that it didnt matter of the scalar
>>> file being returned was stdout/err, or just a plain file. If an array is
>>> returned, nothing else seems to be getting returned correctly.
>>>
>>> Here's the tests:
>>>
>>> swift$ cat amap01.swift
>>> type file;
>>>
>>> app (file a[], file o) retarr()
>>> {
>>> sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" stdout=filename(o);
>>> }
>>>
>>> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
>>> file o<"out.txt">;
>>>
>>> (a,o) = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap01.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run010
>>> Progress: Thu, 29 May 2014 04:44:25+0000
>>>
>>> Execution failed:
>>> Exception in sh:
>>> Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a
>>> >out/a0002.txt; echo I am stdout]
>>> Host: localhost
>>> Directory: amap01-run010/jobs/4/sh-4ipn1brl
>>> stderr.txt:
>>> stdout.txt: I am stdout
>>> exception @ swift-int.k, line: 511
>>> Caused by: The following output files were not created by the
>>> application: out.txt
>>> throw @ swift-int.k, line: 76
>>>
>>> k:assign @ swift.k, line: 194
>>> Caused by: Exception in sh:
>>> Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a
>>> >out/a0002.txt; echo I am stdout]
>>> Host: localhost
>>> Directory: amap01-run010/jobs/4/sh-4ipn1brl
>>> stderr.txt:
>>> stdout.txt: I am stdout
>>> exception @ swift-int.k, line: 511
>>> Caused by: The following output files were not created by the
>>> application: out.txt
>>> throw @ swift-int.k, line: 76
>>>
>>> swift$ cat amap02.swift
>>> type file;
>>>
>>> app (file a, file o) retarr()
>>> {
>>> sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" stdout=filename(o);
>>> }
>>>
>>> file a<"out/a0001.txt">;
>>> file o<"out.txt">;
>>>
>>> (a,o) = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap02.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run011
>>> Progress: Thu, 29 May 2014 04:44:51+0000
>>> Final status:Thu, 29 May 2014 04:44:51+0000 Finished successfully:1
>>> swift$ ls out
>>> a0001.txt
>>> swift$ cat out.txt
>>> I am stdout
>>> swift$ cat amap03.swift
>>> type file;
>>>
>>> app (file a[]) retarr()
>>> {
>>> sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" ;
>>> }
>>>
>>> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
>>>
>>> a = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap03.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run012
>>> Progress: Thu, 29 May 2014 04:45:34+0000
>>> Final status:Thu, 29 May 2014 04:45:34+0000 Finished successfully:1
>>> swift$ ls out
>>> a0001.txt a0002.txt
>>> swift$
>>>
>>>
>>>
>>> On 5/28/14, 11:09 PM, Mihael Hategan wrote:
>>>> Hi Mike and Jonathan,
>>>>
>>>> I'm afraid I need more context. I am unable to reproduce the issue,
>>>> although I did discover another bug in the way FixedArrayMapper parses
>>>> strings.
>>>>
>>>> Can you please tell me what array mapper we are talking about, what the
>>>> app signature is, what provider you are using, and whether provider
>>>> staging is enabled or not.
>>>>
>>>> Better yet, please send me the swift log if you can.
>>>>
>>>> Mihael
>>>>
>>>> On Wed, 2014-05-28 at 18:22 -0500, Michael Wilde wrote:
>>>>> On 5/28/14, 5:01 PM, Jonathan Ozik wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> Has anyone run into issues when trying to direct stdout and stderr to output files and getting the error:
>>>>>> Caused by: The following output files were not created by the application: sim.out, err.out
>>>>> This seems to be caused by an error in Swift (based on investigating
>>>>> with Jonathan off-list).
>>>>>
>>>>> The bug is that if an app returns an array of files, then any non-array
>>>>> returns from the app are not getting handled correctly.
>>>>> It seems to be a problem in the array-collecting code in the Swift
>>>>> app-launching wrapper, which may be loosing its directory context after
>>>>> returning from the collection function.
>>>>>> It looks like the files are being created too late and not caught when the output is being gathered up? This is using trunk, by the way.
>>>>>>
>>>>>> Also, in a related question, is there an easy way to redirect stdout and stderr from an app invocation to the regular swift.out without creating specific files?
>>>>> No, not at the moment. Some variations of this capability have been
>>>>> discussed, though. It seems to me to be a useful addition to implement.
>>>>>
>>>>> - Mike
>>>>>> Jonathan
>>>>>>
>>>>>> _______________________________________________
>>>>>> Swift-user mailing list
>>>>>> Swift-user at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>>
>
> --
> Michael Wilde
> Mathematics and Computer Science Computation Institute
> Argonne National Laboratory The University of Chicago
>
More information about the Swift-user
mailing list