[Swift-user] Known issue with stdout and stderr file delays?

[Mihael Hategan]

Thanks. This was only an issue without provider staging.

Anyway, the problem was that the dynamic data collect code was
overwriting a variable used to point to the directory where the app
output files were. Silly shell scoping rules (or the lack thereof), and
silly me for not seeing it.

Fix committed to trunk.

Mihael

On Wed, 2014-05-28 at 23:50 -0500, Michael Wilde wrote:
> Here's an example of the problem (what looks to me like a Swift bug in 
> the new array return feature).
> 
> amap01.swift tries to return a file array and a single file (stdout). 
> Swift is unable to find the mapped stdout file, even though it exists.
> 
> amap02.swift returns just the stdout file (to show that this part of the 
> test script works).
> 
> amap03.swift returns just the array. It, too, works.
> 
> But returning both together fails, suggesting that the swiftwrap code 
> that collects the array prevents any other non-array files from being 
> returned.
> 
> Tests of Jonathan'sscript indicated that it didnt matter of the scalar 
> file being returned was stdout/err, or just a plain file. If an array is 
> returned, nothing else seems to be getting returned correctly.
> 
> Here's the tests:
> 
> swift$ cat amap01.swift
> type file;
> 
> app (file a[], file o) retarr()
> {
>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
> echo I am stdout" stdout=filename(o);
> }
> 
> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
> file o<"out.txt">;
> 
> (a,o) = retarr();
> swift$ rm -rf out
> swift$ swift amap01.swift
> Swift trunk swift-r7880 cog-r3907
> RunID: run010
> Progress: Thu, 29 May 2014 04:44:25+0000
> 
> Execution failed:
> Exception in sh:
>      Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a 
>  >out/a0002.txt; echo I am stdout]
>      Host: localhost
>      Directory: amap01-run010/jobs/4/sh-4ipn1brl
>      stderr.txt:
>      stdout.txt: I am stdout
>      exception @ swift-int.k, line: 511
> Caused by: The following output files were not created by the 
> application: out.txt
>      throw @ swift-int.k, line: 76
> 
>      k:assign @ swift.k, line: 194
> Caused by: Exception in sh:
>      Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a 
>  >out/a0002.txt; echo I am stdout]
>      Host: localhost
>      Directory: amap01-run010/jobs/4/sh-4ipn1brl
>      stderr.txt:
>      stdout.txt: I am stdout
>      exception @ swift-int.k, line: 511
> Caused by: The following output files were not created by the 
> application: out.txt
>      throw @ swift-int.k, line: 76
> 
> swift$ cat amap02.swift
> type file;
> 
> app (file a, file o) retarr()
> {
>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
> echo I am stdout" stdout=filename(o);
> }
> 
> file a<"out/a0001.txt">;
> file o<"out.txt">;
> 
> (a,o) = retarr();
> swift$ rm -rf out
> swift$ swift amap02.swift
> Swift trunk swift-r7880 cog-r3907
> RunID: run011
> Progress: Thu, 29 May 2014 04:44:51+0000
> Final status:Thu, 29 May 2014 04:44:51+0000  Finished successfully:1
> swift$ ls out
> a0001.txt
> swift$ cat out.txt
> I am stdout
> swift$ cat amap03.swift
> type file;
> 
> app (file a[]) retarr()
> {
>    sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
> echo I am stdout" ;
> }
> 
> file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
> 
> a = retarr();
> swift$ rm -rf out
> swift$ swift amap03.swift
> Swift trunk swift-r7880 cog-r3907
> RunID: run012
> Progress: Thu, 29 May 2014 04:45:34+0000
> Final status:Thu, 29 May 2014 04:45:34+0000  Finished successfully:1
> swift$ ls out
> a0001.txt  a0002.txt
> swift$
> 
> 
> 
> On 5/28/14, 11:09 PM, Mihael Hategan wrote:
> > Hi Mike and Jonathan,
> >
> > I'm afraid I need more context. I am unable to reproduce the issue,
> > although I did discover another bug in the way FixedArrayMapper parses
> > strings.
> >
> > Can you please tell me what array mapper we are talking about, what the
> > app signature is, what provider you are using, and whether provider
> > staging is enabled or not.
> >
> > Better yet, please send me the swift log if you can.
> >
> > Mihael
> >
> > On Wed, 2014-05-28 at 18:22 -0500, Michael Wilde wrote:
> >> On 5/28/14, 5:01 PM, Jonathan Ozik wrote:
> >>> Hi all,
> >>>
> >>> Has anyone run into issues when trying to direct stdout and stderr to output files and getting the error:
> >>> Caused by: The following output files were not created by the application: sim.out, err.out
> >> This seems to be caused by an error in Swift (based on investigating
> >> with Jonathan off-list).
> >>
> >> The bug is that if an app returns an array of files, then any non-array
> >> returns from the app are not getting handled correctly.
> >> It seems to be a problem in the array-collecting code in the Swift
> >> app-launching wrapper, which may be loosing its directory context after
> >> returning from the collection function.
> >>> It looks like the files are being created too late and not caught when the output is being gathered up? This is using trunk, by the way.
> >>>
> >>> Also, in a related question, is there an easy way to redirect stdout and stderr from an app invocation to the regular swift.out without creating specific files?
> >> No, not at the moment. Some variations of this capability have been
> >> discussed, though. It seems to me to be a useful addition to implement.
> >>
> >> - Mike
> >>> Jonathan
> >>>
> >>> _______________________________________________
> >>> Swift-user mailing list
> >>> Swift-user at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
>