[Swift-devel] jobthrottle value does not correspond to number of parallel jobs on local provider

[Michael Wilde]

Ketan,

I tested this with a recent trunk rev and it seems to work. Can you try to replicate the test below and proceed from there to make your specific app pattern work?

This was run on pads.

- Mike


login2$ swift -config cf -cdm.file direct -tc.file tc -sites.file sites.xml catsndirect.swift -n=1

Swift trunk swift-r5920 cog-r3471 (cog modified locally)

RunID: 20121023-1554-vese3gh6
Progress:  time: Tue, 23 Oct 2012 15:54:47 -0500
Final status: Tue, 23 Oct 2012 15:54:47 -0500  Finished successfully:1

login2$ cat catsndirect.swift
type file;

app (file o) cat (file i)
{
  cat @i stdout=@o;
}

file out[]<simple_mapper; location="/tmp/wilde/outdir", prefix="f.",suffix=".out">;

foreach j in [1:@toint(@arg("n","1"))] {
  file data<"/tmp/wilde/indir/data.txt">;
  out[j] = cat(data);
}

login2$ cat cf
wrapperlog.always.transfer=true
sitedir.keep=true
execution.retries=0
lazy.errors=false
status.mode=provider
use.provider.staging=false
provider.staging.pin.swiftfiles=false
use.wrapper.staging=false

login2$ cat direct
rule  .* DIRECT /

login2$ cat sites.xml
<config>
  <pool handle="localhost">
    <execution provider="local" />
    <filesystem provider="local" />
    <workdirectory>/scratch/local/wilde/pstest/swiftwork</workdirectory>
  </pool>
</config>
login2$

----- Original Message -----
> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Tuesday, October 23, 2012 3:23:16 PM
> Subject: Re: [Swift-devel] jobthrottle value does not correspond to number of parallel jobs on local provider
> I tried CDM DIRECT policy with catsnsleep which gives the following:
> 
> 
> CDM[DIRECT]: Linking to /home/train07/ketan_mars/swift/data.txt
> failed!
> 
> 
> the file exists as mentioned in the path above.
> 
> 
> When specifying relative path to data.txt, I get /data.txt doesn't
> exist, presumably because cdm DIRECT assumes paths to be absolute.
> 
> 
> Log attached.
> 
> 
> On Tue, Oct 23, 2012 at 4:14 PM, Michael Wilde < wilde at mcs.anl.gov >
> wrote:
> 
> 
> 
> > Now trying with cdm. My cdm policy file contains a single line as
> > follows:
> >
> >
> > rule .* DEFAULT /
> 
> Change DEFAULT to DIRECT
> 
> Look at the example at:
> 
> http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_specific_use_cases
> 
> sec 20.5.1 and 20.5.2
> 
> Its best to test this first with simple "catsn-like" examples before
> you try your science app with it, to make sure that the direct
> processing is behaving as you expect.
> 
> - Mike
> 
> 
> 
> 
> --
> Ketan

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory