[Swift-user] Known issue with stdout and stderr file delays?

Jonathan Ozik jozik at uchicago.edu
Thu May 29 09:51:46 CDT 2014


Thank you Mihael. These fixes will be very helpful.

Jonathan

On May 29, 2014, at 9:46 AM, Michael Wilde <wilde at anl.gov> wrote:

> Thanks, Mihael. My tests for the two bugs related to app array return mapping now work, on trunk.
> 
> 1) returning scalar files along with array returns from the same app
> 2) returning output arrays mapped with simple_mapper without location= argument
> 
> Yadu, these tests are on midway in ~wilde/swift/lab/arrmapbug/*swift
> 
> - Mike
> 
> 
> On 5/29/14, 4:26 AM, Mihael Hategan wrote:
>> Thanks. This was only an issue without provider staging.
>> 
>> Anyway, the problem was that the dynamic data collect code was
>> overwriting a variable used to point to the directory where the app
>> output files were. Silly shell scoping rules (or the lack thereof), and
>> silly me for not seeing it.
>> 
>> Fix committed to trunk.
>> 
>> Mihael
>> 
>> On Wed, 2014-05-28 at 23:50 -0500, Michael Wilde wrote:
>>> Here's an example of the problem (what looks to me like a Swift bug in
>>> the new array return feature).
>>> 
>>> amap01.swift tries to return a file array and a single file (stdout).
>>> Swift is unable to find the mapped stdout file, even though it exists.
>>> 
>>> amap02.swift returns just the stdout file (to show that this part of the
>>> test script works).
>>> 
>>> amap03.swift returns just the array. It, too, works.
>>> 
>>> But returning both together fails, suggesting that the swiftwrap code
>>> that collects the array prevents any other non-array files from being
>>> returned.
>>> 
>>> Tests of Jonathan'sscript indicated that it didnt matter of the scalar
>>> file being returned was stdout/err, or just a plain file. If an array is
>>> returned, nothing else seems to be getting returned correctly.
>>> 
>>> Here's the tests:
>>> 
>>> swift$ cat amap01.swift
>>> type file;
>>> 
>>> app (file a[], file o) retarr()
>>> {
>>>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" stdout=filename(o);
>>> }
>>> 
>>> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
>>> file o<"out.txt">;
>>> 
>>> (a,o) = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap01.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run010
>>> Progress: Thu, 29 May 2014 04:44:25+0000
>>> 
>>> Execution failed:
>>> Exception in sh:
>>>      Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a
>>>  >out/a0002.txt; echo I am stdout]
>>>      Host: localhost
>>>      Directory: amap01-run010/jobs/4/sh-4ipn1brl
>>>      stderr.txt:
>>>      stdout.txt: I am stdout
>>>      exception @ swift-int.k, line: 511
>>> Caused by: The following output files were not created by the
>>> application: out.txt
>>>      throw @ swift-int.k, line: 76
>>> 
>>>      k:assign @ swift.k, line: 194
>>> Caused by: Exception in sh:
>>>      Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a
>>>  >out/a0002.txt; echo I am stdout]
>>>      Host: localhost
>>>      Directory: amap01-run010/jobs/4/sh-4ipn1brl
>>>      stderr.txt:
>>>      stdout.txt: I am stdout
>>>      exception @ swift-int.k, line: 511
>>> Caused by: The following output files were not created by the
>>> application: out.txt
>>>      throw @ swift-int.k, line: 76
>>> 
>>> swift$ cat amap02.swift
>>> type file;
>>> 
>>> app (file a, file o) retarr()
>>> {
>>>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" stdout=filename(o);
>>> }
>>> 
>>> file a<"out/a0001.txt">;
>>> file o<"out.txt">;
>>> 
>>> (a,o) = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap02.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run011
>>> Progress: Thu, 29 May 2014 04:44:51+0000
>>> Final status:Thu, 29 May 2014 04:44:51+0000  Finished successfully:1
>>> swift$ ls out
>>> a0001.txt
>>> swift$ cat out.txt
>>> I am stdout
>>> swift$ cat amap03.swift
>>> type file;
>>> 
>>> app (file a[]) retarr()
>>> {
>>>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt;
>>> echo I am stdout" ;
>>> }
>>> 
>>> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
>>> 
>>> a = retarr();
>>> swift$ rm -rf out
>>> swift$ swift amap03.swift
>>> Swift trunk swift-r7880 cog-r3907
>>> RunID: run012
>>> Progress: Thu, 29 May 2014 04:45:34+0000
>>> Final status:Thu, 29 May 2014 04:45:34+0000  Finished successfully:1
>>> swift$ ls out
>>> a0001.txt  a0002.txt
>>> swift$
>>> 
>>> 
>>> 
>>> On 5/28/14, 11:09 PM, Mihael Hategan wrote:
>>>> Hi Mike and Jonathan,
>>>> 
>>>> I'm afraid I need more context. I am unable to reproduce the issue,
>>>> although I did discover another bug in the way FixedArrayMapper parses
>>>> strings.
>>>> 
>>>> Can you please tell me what array mapper we are talking about, what the
>>>> app signature is, what provider you are using, and whether provider
>>>> staging is enabled or not.
>>>> 
>>>> Better yet, please send me the swift log if you can.
>>>> 
>>>> Mihael
>>>> 
>>>> On Wed, 2014-05-28 at 18:22 -0500, Michael Wilde wrote:
>>>>> On 5/28/14, 5:01 PM, Jonathan Ozik wrote:
>>>>>> Hi all,
>>>>>> 
>>>>>> Has anyone run into issues when trying to direct stdout and stderr to output files and getting the error:
>>>>>> Caused by: The following output files were not created by the application: sim.out, err.out
>>>>> This seems to be caused by an error in Swift (based on investigating
>>>>> with Jonathan off-list).
>>>>> 
>>>>> The bug is that if an app returns an array of files, then any non-array
>>>>> returns from the app are not getting handled correctly.
>>>>> It seems to be a problem in the array-collecting code in the Swift
>>>>> app-launching wrapper, which may be loosing its directory context after
>>>>> returning from the collection function.
>>>>>> It looks like the files are being created too late and not caught when the output is being gathered up? This is using trunk, by the way.
>>>>>> 
>>>>>> Also, in a related question, is there an easy way to redirect stdout and stderr from an app invocation to the regular swift.out without creating specific files?
>>>>> No, not at the moment. Some variations of this capability have been
>>>>> discussed, though. It seems to me to be a useful addition to implement.
>>>>> 
>>>>> - Mike
>>>>>> Jonathan
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Swift-user mailing list
>>>>>> Swift-user at ci.uchicago.edu
>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> 
> 
> -- 
> Michael Wilde
> Mathematics and Computer Science          Computation Institute
> Argonne National Laboratory               The University of Chicago
> 




More information about the Swift-user mailing list