[Swift-devel] problems with external dependencies

[Michael Wilde]

On 3/23/09 3:51 PM, Ben Clifford wrote:
> ok I'll have a play with the below. But not until you send me the log file 
> where the code I sent you doesn't work for you but does work for me.

deal. http://www.ci.uchicag.edu:~wilde/ben-20090323-2222-ubxsbh83.log

and below.

Did I run the right version of your test?

close but not the latest trunk. I will update and see if it makes a diff.

I get:

sur$ rm ben.xml ben.kml
sur$
sur$ cat ben.swift
type file;

app (file o) echo (int i) { echo i stdout=@o; }

(external s, file r[]) generate() { 

   int j[] = [0:10];
   foreach i in j {
     r[i] = echo(i*i);
   }
}

app (file o) ls (string dir, external w) { ls "-l" dir stdout=@o; }

file datadir[]<simple_mapper;prefix="datadir/">;
external sync;
(sync,datadir) = generate();

file out <"ls.out">;
out = ls("datadir/", sync);
sur$ rm -rf datadir ls.out
sur$ ls -l datadir ls.out
/bin/ls: datadir: No such file or directory
/bin/ls: ls.out: No such file or directory
sur$ swift ben.swift
Swift svn swift-r2724 (swift modified locally) cog-r2333

RunID: 20090323-2222-ubxsbh83
Progress:
Progress:  Selecting site:8 Stage in:1 Finished successfully:2
Progress:  Selecting site:2 Submitted:1 Finished successfully:8
getValue called in an external dataset
Final status:  Failed:1 Finished successfully:11
The following errors have occurred:
1. Application "ls" failed (Job failed with an exit code of 2)
         Arguments: "-l, datadir/"
         Host: localhost
         Directory: ben-20090323-2222-ubxsbh83/jobs/s/ls-sheljd8j
         STDERR: /bin/ls: datadir/: No such file or directory
         STDOUT:
sur$


> On Mon, 23 Mar 2009, Michael Wilde wrote:
> 
>> On 3/23/09 2:18 PM, Ben Clifford wrote:
>>> On Mon, 23 Mar 2009, Michael Wilde wrote:
>>>
>>>> Passing 2000 outputs from the simuation to the analysis routine causes
>>>> wrapper.sh to be invoked with values of DIRS and/or INF that exceed the
>>>> max
>>>> allowed command line length (which I think is 128KB on the bgp).
>>>>
>>>> Its primarily the error causes by this length limit that Im trying to work
>>>> around here.
>>> It might not be too hard to make wrapper.sh take its inputs from a file,
>>> which would then scale much better. I've certainly thought about it before.
>> Cool. Can you do the best-of-both-worlds, passing short to modest size lists
>> as we do today, and only using files for INF and DIRS when they are very long?
>>
>> Eg, if the first char of dirs is ` as in `listofdirs` then wrapper.sh reads
>> them from file "listofdirs". Otherwise, I would fear that we're stepping
>> backwards on the path of eliminating file operations in the app-calling path.
>>
>>> Would your app be able to cope ok if wrapper.sh did not have this problem?
>> Yes, I think so, but surprises are possible.
>>
>>> (or would it also have the same command-line length problems?)
>> No, it avoided command-line length limitations from the start by only placing
>> the first file name in the struct-array on the command line, to serve as a
>> pattern for the script, which knows how to find the full set of files, by
>> naming convention.
>>
>>
>>
>>