[Swift-user] Known issue with stdout and stderr file delays?

[Michael Wilde]

Here's an example of the problem (what looks to me like a Swift bug in 
the new array return feature).

amap01.swift tries to return a file array and a single file (stdout). 
Swift is unable to find the mapped stdout file, even though it exists.

amap02.swift returns just the stdout file (to show that this part of the 
test script works).

amap03.swift returns just the array. It, too, works.

But returning both together fails, suggesting that the swiftwrap code 
that collects the array prevents any other non-array files from being 
returned.

Tests of Jonathan'sscript indicated that it didnt matter of the scalar 
file being returned was stdout/err, or just a plain file. If an array is 
returned, nothing else seems to be getting returned correctly.

Here's the tests:

swift$ cat amap01.swift
type file;

app (file a[], file o) retarr()
{
   sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
echo I am stdout" stdout=filename(o);
}

file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;
file o<"out.txt">;

(a,o) = retarr();
swift$ rm -rf out
swift$ swift amap01.swift
Swift trunk swift-r7880 cog-r3907
RunID: run010
Progress: Thu, 29 May 2014 04:44:25+0000

Execution failed:
Exception in sh:
     Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a 
 >out/a0002.txt; echo I am stdout]
     Host: localhost
     Directory: amap01-run010/jobs/4/sh-4ipn1brl
     stderr.txt:
     stdout.txt: I am stdout
     exception @ swift-int.k, line: 511
Caused by: The following output files were not created by the 
application: out.txt
     throw @ swift-int.k, line: 76

     k:assign @ swift.k, line: 194
Caused by: Exception in sh:
     Arguments: [-c, mkdir -p out; echo a >out/a0001.txt; echo a 
 >out/a0002.txt; echo I am stdout]
     Host: localhost
     Directory: amap01-run010/jobs/4/sh-4ipn1brl
     stderr.txt:
     stdout.txt: I am stdout
     exception @ swift-int.k, line: 511
Caused by: The following output files were not created by the 
application: out.txt
     throw @ swift-int.k, line: 76

swift$ cat amap02.swift
type file;

app (file a, file o) retarr()
{
   sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
echo I am stdout" stdout=filename(o);
}

file a<"out/a0001.txt">;
file o<"out.txt">;

(a,o) = retarr();
swift$ rm -rf out
swift$ swift amap02.swift
Swift trunk swift-r7880 cog-r3907
RunID: run011
Progress: Thu, 29 May 2014 04:44:51+0000
Final status:Thu, 29 May 2014 04:44:51+0000  Finished successfully:1
swift$ ls out
a0001.txt
swift$ cat out.txt
I am stdout
swift$ cat amap03.swift
type file;

app (file a[]) retarr()
{
   sh "-c" "mkdir -p out; echo a >out/a0001.txt; echo a >out/a0002.txt; 
echo I am stdout" ;
}

file a[]<simple_mapper; location="out", prefix="a", suffix=".txt">;

a = retarr();
swift$ rm -rf out
swift$ swift amap03.swift
Swift trunk swift-r7880 cog-r3907
RunID: run012
Progress: Thu, 29 May 2014 04:45:34+0000
Final status:Thu, 29 May 2014 04:45:34+0000  Finished successfully:1
swift$ ls out
a0001.txt  a0002.txt
swift$



On 5/28/14, 11:09 PM, Mihael Hategan wrote:
> Hi Mike and Jonathan,
>
> I'm afraid I need more context. I am unable to reproduce the issue,
> although I did discover another bug in the way FixedArrayMapper parses
> strings.
>
> Can you please tell me what array mapper we are talking about, what the
> app signature is, what provider you are using, and whether provider
> staging is enabled or not.
>
> Better yet, please send me the swift log if you can.
>
> Mihael
>
> On Wed, 2014-05-28 at 18:22 -0500, Michael Wilde wrote:
>> On 5/28/14, 5:01 PM, Jonathan Ozik wrote:
>>> Hi all,
>>>
>>> Has anyone run into issues when trying to direct stdout and stderr to output files and getting the error:
>>> Caused by: The following output files were not created by the application: sim.out, err.out
>> This seems to be caused by an error in Swift (based on investigating
>> with Jonathan off-list).
>>
>> The bug is that if an app returns an array of files, then any non-array
>> returns from the app are not getting handled correctly.
>> It seems to be a problem in the array-collecting code in the Swift
>> app-launching wrapper, which may be loosing its directory context after
>> returning from the collection function.
>>> It looks like the files are being created too late and not caught when the output is being gathered up? This is using trunk, by the way.
>>>
>>> Also, in a related question, is there an easy way to redirect stdout and stderr from an app invocation to the regular swift.out without creating specific files?
>> No, not at the moment. Some variations of this capability have been
>> discussed, though. It seems to me to be a useful addition to implement.
>>
>> - Mike
>>> Jonathan
>>>
>>> _______________________________________________
>>> Swift-user mailing list
>>> Swift-user at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago