[Swift-devel] passing very long lists of files to applications

[Ben Clifford]

Some applications have a problem where a component program is to process a 
very large number of files.

Swift can deal with files in two basic ways at the moment:

 i) explicitly pass the filenames on the commandline
 ii) stage the files into the job input directory without explicitly 
     naming the files on the commandline, with the component program 
     inspecting the working directory on the execution side to decide 
     which files to process.

i) has the disadvantage that the commandline limits the number of 
filenames that can be passed

ii) has the disadvantage that the component program must be able to 
distinguish which of their working directory files are the relevant input 
files.

A further option which could be implemented is to provide the ability to 
write out a list of filenames into a file, and have that file staged as 
input. This needs the component program to be able to take a list of files 
from a file rather than from the command line (for example, the -T option 
of tar).

This could be implemented, I think, by providing a writeData procedure 
which is the inverse of readData, and writing something like this:

file l = writeData(@filenames(f))
p(l,f);

app p(file l, file f[]) {
  myproc "-T" @l
}

comments?

--