[Swift-devel] Problems Re: Example of wrapper staging?

[Michael Wilde]

Below is a simpler example of the failing case:

$ hostname
midway001

# Works with echo as app and wrapperStaging:

$ cat t03.swift
type file;

app (file o) echo () {
     echo "hello" stdout=filename(o);
}

file o <"out.txt">;

o = echo();
$ cat swift.conf
wrapperStagingLocalServer: "file://"

app.sh { executable: /bin/sh, maxWallTime: "00:01:00" }

site.local {
     staging: "wrapper"
     scratch: "/tmp/wilde/swiftest"
     workDirectory: "/tmp/wilde/swiftwork"
}

lazyErrors: false
executionRetries: 0
keepSiteDir: true
alwaysTransferWrapperLog: true


$ rm out.txt

# Fails with sh as app and wrapper staging:

$ swift -config swift.conf t03.swift
Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 
heads/master 6172
RunID: run011
Progress: Sun, 09 Nov 2014 20:20:39+0000
Final status:Sun, 09 Nov 2014 20:20:40+0000  Finished successfully:1
$ cat out.txt
hello
$ cat t04.swift
type file;

app (file o) echo () {
     sh "-c" "/bin/echo hello" stdout=filename(o);
}

file o <"out.txt">;

o = echo();
$ swift -config swift.conf t04.swift
Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 
heads/master 6172
RunID: run012
Progress: Sun, 09 Nov 2014 20:21:09+0000
Final status:Sun, 09 Nov 2014 20:21:09+0000  Finished successfully:1
$ cat out.txt

$ ls -l out.txt
-rw-rw-r-- 1 wilde wilde 1 Nov  9 14:21 out.txt
$ od -x out.txt
0000000 000a
0000001

# Works with sh as app and default staging:

$ swift -config swift.conf.nows t04.swift
Swift trunk git-rev: 7a0c4927380514e4c7166dfd095fd0a4c6389b03 
heads/master 6172
RunID: run013
Progress: Sun, 09 Nov 2014 20:21:59+0000
Final status:Sun, 09 Nov 2014 20:21:59+0000  Finished successfully:1
$ cat swift.conf.nows
wrapperStagingLocalServer: "file://"

app.sh { executable: /bin/sh, maxWallTime: "00:01:00" }

lazyErrors: false
executionRetries: 0

$ cat out.txt
hello
$

On 11/9/14 2:05 PM, Michael Wilde wrote:
> Oddly, when I use "sh" -> /bin/sh for the app, this example fails.  The
> swift run succeeds, but the returned file is zero-length (for the
> example below) or contains a single newline (when I test a sh -c of a
> simple echo).
>
> It *seems* that using cat as the app works while /bin/sh fails to return
> output correctly.
>
> I'll need to debug further, but below are the working and then failing
> swift scripts:
>
> # This works, c.txt comes back as 1M+12 bytes:
>
> $ cat wrapstage.swift
> type file;
>
> app (file c) cat(file a, file b) {
>       cat filename(a) filename(b) stdout=filename(c);
> }
>
> file c <"c.txt">;
> file a <"a.txt">;
> file b <"http://mirror.internode.on.net/pub/test/1meg.test">;
>
> c = cat(a, b);
>
> # this fails: c.txt comes back as zero length:
>
> $ cat wrapstage2.swift
> type file;
>
> app (file c) cat(file a, file b) {
>       sh "-c" strcat("/bin/cat ", filename(a), " ", filename(b))
> stdout=filename(c);
> }
>
> file c <"c.txt">;
> file a <"a.txt">;
> file b <"http://mirror.internode.on.net/pub/test/1meg.test">;
>
> c = cat(a, b);
> swift$
>
> A bug, or user error?
>
> - Mike
>
> On 11/8/14 5:32 PM, Michael Wilde wrote:
>> Ok, that works great now, both with the conf you posted and with this one:
>>
>> wrapperStagingLocalServer: "file://"
>>
>> site.local {
>>        staging: "wrapper"
>>        scratch: "/tmp/wilde/swiftest"
>> }
>>
>> Thanks,
>>
>> - Mike
>>
>> On 11/8/14 4:41 PM, Mihael Hategan wrote:
>>> (solved on IM but replying here for reference).
>>>
>>> I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is
>>> a global option not a site option. However, it should work with the
>>> vanilla swift.conf posted, which had the wrong option commented out.
>>>
>>> Mihael
>>>
>>>
>>> On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote:
>>>> I get this error:
>>>>
>>>> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected
>>>> property 'site.local.wrapperStagingLocalServer'
>>>>         at
>>>> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214)
>>>>
>>>> Ive tried spelling this property both as in your sample .conf and as in
>>>> your text:
>>>>
>>>> wrapperStagingLocalServer and wrapperStageingLocalServer
>>>>
>>>> Neither is accepted.  Did a schema change not get committed?
>>>>
>>>> For general info: where is the json schema file that a user can check to
>>>> find all valid conf parameters?
>>>>
>>>> Thanks,
>>>>
>>>> - Mike
>>>>
>>>>
>>>>
>>>> On 11/8/14 4:13 PM, Mihael Hategan wrote:
>>>>> type file;
>>>>>
>>>>> app (file c) cat(file a, file b) {
>>>>> 	cat filename(a) filename(b) stdout=filename(c);
>>>>> }
>>>>>
>>>>> file c <"c.txt">;
>>>>> file a <"a.txt">;
>>>>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">;
>>>>>
>>>>> c = cat(a, b);
>>>>> --------------------------------------------------------
>>>>>
>>>>> You would need a proper swift.conf. Here's an example of that:
>>>>> --------------------------------------------------------
>>>>> site.local {
>>>>> 	staging: "wrapper"
>>>>> 	scratch: "/some/scratch/dir"
>>>>>             # *see email comments for details on the following option
>>>>> 	# wrapperStagingLocalServer:"file://"
>>>>> }
>>>>> --------------------------------------------------------
>>>>>
>>>>>
>>>>> With a proper scratch dir there, this*should*  work for you without any
>>>>> additional tweaking.
>>>>>
>>>>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions.
>>>>> That's where you would add additional protocols.
>>>>>
>>>>> The wrapperStagingLocalServer sets the default staging location for
>>>>> things that do not have an explicit scheme. In the case above, "c.txt"
>>>>> and "a.txt" are staged in and out using"file://". If you wanted the
>>>>> default to be some gridftp server, you would say something like
>>>>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course,
>>>>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs,
>>>>> which it doesn't at the moment.
>>>>>
>>>>> Mihael

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago