[Swift-devel] Problems Re: Example of wrapper staging?

[Michael Wilde]

Oddly, when I use "sh" -> /bin/sh for the app, this example fails.  The 
swift run succeeds, but the returned file is zero-length (for the 
example below) or contains a single newline (when I test a sh -c of a 
simple echo).

It *seems* that using cat as the app works while /bin/sh fails to return 
output correctly.

I'll need to debug further, but below are the working and then failing 
swift scripts:

# This works, c.txt comes back as 1M+12 bytes:

$ cat wrapstage.swift
type file;

app (file c) cat(file a, file b) {
     cat filename(a) filename(b) stdout=filename(c);
}

file c <"c.txt">;
file a <"a.txt">;
file b <"http://mirror.internode.on.net/pub/test/1meg.test">;

c = cat(a, b);

# this fails: c.txt comes back as zero length:

$ cat wrapstage2.swift
type file;

app (file c) cat(file a, file b) {
     sh "-c" strcat("/bin/cat ", filename(a), " ", filename(b)) 
stdout=filename(c);
}

file c <"c.txt">;
file a <"a.txt">;
file b <"http://mirror.internode.on.net/pub/test/1meg.test">;

c = cat(a, b);
swift$

A bug, or user error?

- Mike

On 11/8/14 5:32 PM, Michael Wilde wrote:
> Ok, that works great now, both with the conf you posted and with this one:
>
> wrapperStagingLocalServer: "file://"
>
> site.local {
>       staging: "wrapper"
>       scratch: "/tmp/wilde/swiftest"
> }
>
> Thanks,
>
> - Mike
>
> On 11/8/14 4:41 PM, Mihael Hategan wrote:
>> (solved on IM but replying here for reference).
>>
>> I made a mistake in the swift.conf I sent. wrapperStagingLocalServer is
>> a global option not a site option. However, it should work with the
>> vanilla swift.conf posted, which had the wrong option commented out.
>>
>> Mihael
>>
>>
>> On Sat, 2014-11-08 at 16:37 -0600, Michael Wilde wrote:
>>> I get this error:
>>>
>>> org.griphyn.vdl.util.SwiftConfigException: swift.conf:4 unexpected
>>> property 'site.local.wrapperStagingLocalServer'
>>>        at
>>> org.griphyn.vdl.util.SwiftConfigSchema.validate(SwiftConfigSchema.java:214)
>>>
>>> Ive tried spelling this property both as in your sample .conf and as in
>>> your text:
>>>
>>> wrapperStagingLocalServer and wrapperStageingLocalServer
>>>
>>> Neither is accepted.  Did a schema change not get committed?
>>>
>>> For general info: where is the json schema file that a user can check to
>>> find all valid conf parameters?
>>>
>>> Thanks,
>>>
>>> - Mike
>>>
>>>
>>>
>>> On 11/8/14 4:13 PM, Mihael Hategan wrote:
>>>> type file;
>>>>
>>>> app (file c) cat(file a, file b) {
>>>> 	cat filename(a) filename(b) stdout=filename(c);
>>>> }
>>>>
>>>> file c <"c.txt">;
>>>> file a <"a.txt">;
>>>> file b <"http://mirror.internode.on.net/pub/test/1meg.test">;
>>>>
>>>> c = cat(a, b);
>>>> --------------------------------------------------------
>>>>
>>>> You would need a proper swift.conf. Here's an example of that:
>>>> --------------------------------------------------------
>>>> site.local {
>>>> 	staging: "wrapper"
>>>> 	scratch: "/some/scratch/dir"
>>>>            # *see email comments for details on the following option
>>>> 	# wrapperStagingLocalServer:"file://"
>>>> }
>>>> --------------------------------------------------------
>>>>
>>>>
>>>> With a proper scratch dir there, this*should*  work for you without any
>>>> additional tweaking.
>>>>
>>>> Check _swiftwrap.wrapperstaging's stagein() and stageout() functions.
>>>> That's where you would add additional protocols.
>>>>
>>>> The wrapperStagingLocalServer sets the default staging location for
>>>> things that do not have an explicit scheme. In the case above, "c.txt"
>>>> and "a.txt" are staged in and out using"file://". If you wanted the
>>>> default to be some gridftp server, you would say something like
>>>> 'wrapperStageingLocalServer: "gsiftp://gridftp.server.org/"'. Of course,
>>>> _swiftwrap.wrapperstaging would need to know how to handle gsiftp URLs,
>>>> which it doesn't at the moment.
>>>>
>>>> Mihael

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago