[Swift-user] Argument list to long

[jon.monette at gmail.com]

What do you mean by manage them independently?
Sent on the Sprint® Now Network from my BlackBerry®

-----Original Message-----
From: Mihael Hategan <hategan at mcs.anl.gov>
Date: Sat, 09 Oct 2010 12:58:13 
To: Jonathan Monette<jon.monette at gmail.com>
Cc: Michael Wilde<wilde at mcs.anl.gov>; <swift-user at ci.uchicago.edu>
Subject: Re: [Swift-user] Argument list to long

So swift will still pass all elements in imgs[] to the wrapper as the
list of input files, and try to stage them in.

If you have them locally and you are managing them independently, you
may want to try to define them as "external".

As for the parameter files, I have a suspicion that the following line
(taken from one such parameter file) may be responsible for the
problems:

-scratch-e /home/jonmon/Library/Montage/bin/mProject

(each parameter is supposed to be on its own line).

I'll see if I can fix that.

On Sat, 2010-10-09 at 14:47 -0500, Jonathan Monette wrote:
> app ( Table img_tbl ) mImgtbl( Image imgs[] )
> {
>      mImgtbl @dirname( imgs[0] ) @img_tbl;
> }
> 
> On 10/9/10 2:45 PM, Mihael Hategan wrote:
> > Can you post the app procedure from your swift file that invokes
> > mImgtbl?
> >
> > Mihael
> >
> > On Sat, 2010-10-09 at 13:44 -0500, Jonathan Monette wrote:
> >> My bad.  Forgot to put them in the home directory.  They are there now
> >>
> >> On 10/9/10 1:41 PM, Mihael Hategan wrote:
> >>> And these files would be where exactly?
> >>>
> >>> On Mon, 2010-10-04 at 21:48 -0500, Jonathan Monette wrote:
> >>>> Sure can.  I have two log files in my home directory on the ci
> >>>> machines.  ~jonmon
> >>>> One is rather large(20M) and that is the one with
> >>>> "wrapper.paramter.mode=files" not set.  The other is about 4.7M and that
> >>>> one has "wrapper.parameter.mode=files" set.
> >>>> You should have permissions to read them.
> >>>>
> >>>> On 10/04/2010 09:44 PM, Mihael Hategan wrote:
> >>>>> Ok. I believe that now you hit a bug in swift. Luckily there might be
> >>>>> something we can do about that.
> >>>>> May I have the log file?
> >>>>>
> >>>>> On Mon, 2010-10-04 at 21:39 -0500, Jonathan Monette wrote:
> >>>>>> By setting "wrapper.parameter.mode=files" I get "failed to transfer
> >>>>>> wrapper logs".  Here is my swift.properties file.
> >>>>>>
> >>>>>> execution.retries=0
> >>>>>> sitedir.keep=true
> >>>>>> status.mode=provider
> >>>>>> //wrapper.log.always.transfer=true
> >>>>>> foreach.maxthreads=1024
> >>>>>> wrapper.parameter.mode=files
> >>>>>>
> >>>>>> I have tried this with "wrapper.log.always.transfer=true" both commented
> >>>>>> and uncommented still get the same error.
> >>>>>>
> >>>>>> On 10/04/2010 08:25 PM, Mihael Hategan wrote:
> >>>>>>> Groovy. Then set "wrapper.parameter.mode=files" in swift.properties.
> >>>>>>>
> >>>>>>> On Mon, 2010-10-04 at 18:59 -0500, Jonathan Monette wrote:
> >>>>>>>> well if I am understanding the problem the input list will be about
> >>>>>>>> 4100 files and the output list will be a single file (unless Swift adds
> >>>>>>>> more input and output files).  I do not think I am using provider
> >>>>>>>> staging though.  In my swift.properties i do not set the
> >>>>>>>> use.provider.staging option and in etc/swift.properties
> >>>>>>>> use.provider.staging is set to false.
> >>>>>>>>
> >>>>>>>> On 10/4/10 6:46 PM, Mihael Hategan wrote:
> >>>>>>>>> On Mon, 2010-10-04 at 18:39 -0500, Jonathan Monette wrote:
> >>>>>>>>>> Yes.  I have to make sure that all 4100 files are created before the
> >>>>>>>>>> failing app can execute.
> >>>>>>>>>>
> >>>>>>>>>> The writeData() way seems like a hack to get around the problem is Swift
> >>>>>>>>>> but I will try it out and see if my script completes.
> >>>>>>>>>>
> >>>>>>>>>> Mihael: pasted below is the stack trace that was generated in the log file.
> >>>>>>>>> Yeah. It's what Mike says.
> >>>>>>>>>
> >>>>>>>>> But it's not the app arguments. Instead, I'm guessing it's the
> >>>>>>>>> input/output file lists.
> >>>>>>>>>
> >>>>>>>>> There was a scheme in non-provider-staging swift to pass these things in
> >>>>>>>>> lists, but I'm guessing you are using provider staging. Perhaps some
> >>>>>>>>> mode to automatically do this for large numbers of arguments is in
> >>>>>>>>> order.
> >>>>>>>>>
> >>>>>>>>> Mihael
> >>>>>>>>>
> >
>