[Swift-user] Advice on mapping input arguments

[Michael Wilde]

Hi TJ,

Here's a quick initial thought. There might be better ways to do this.

The problem as you mention is that there is currently no mapper or function that lets you easily handle a mixture of scalars and files in a struct.  I think that readData and/or mappers should/could handle this, but Ive been outvoted in past discussions of this.

So for now, assuming you have more scalar params than file params, you can just leave the filenames as strings in the parameter struct, then map the smaller number of files from these strings, and call your app() using the param struct to pass all the scalar params, and pass the mapped files as additional input and output parameters.

This retains, I think, the benefit of having a parameter file that provides a handy record of the run's parameters.

Here's an initial example that I *think* is in the spirit of the longer one you supplied below.

If this sounds like a reasonable approach I think we can use the same technique to run your example.

I provide I think all the files you need to test this.

I'll look more carefully at your example now to see if this is indeed what you're trying to do.

- Mike

mid$ cat params.txt
p1 p2 fn1    fn2    ofn
86 99 f1.dat f2.dat row1.out
87 98 f3.dat f4.dat row2.out

mid$ cat structs.swift

type file;

type params {
  int p1;
  int p2;
  string fn1;
  string fn2;
  string ofn;
};

app (file out) echo (params p, file f1, file f2)
{
  echo stdout=@out "params:" p.p1 p.p2 "files:" @f1 @f2;
}

params plist[] = readData("params.txt");

foreach p, i in plist {
  file f1 <single_file_mapper; file=p.fn1>;
  file f2 <single_file_mapper; file=p.fn2>;
  file of <single_file_mapper; file=p.ofn>;
  of = echo(p,f1,f2);
}

mid$ cat ~/.swift/swift.properties

sites.file=sites.xml
tc.file=apps

status.mode=provider
use.provider.staging=false
use.wrapper.staging=false
wrapperlog.always.transfer=true
execution.retries=0
lazy.errors=false
provider.staging.pin.swiftfiles=false
sitedir.keep=true
file.gc.enabled=false
#tcp.port.range=50000,51000

mid$ cat apps
localhost echo echo

mid$ cat sites.xml
<config>
  <pool handle="localhost">
    <execution provider="local"/>
    <filesystem provider="local"/>
    <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
  </pool>
</config>

mid$ ls -l f?.dat
-rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f1.dat
-rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f2.dat
-rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f3.dat
-rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f4.dat

mid$ swift structs.swift
Swift 0.94 swift-r6414 (swift modified locally) cog-r3648

RunID: 20130611-0343-so454ho1
Progress:  time: Tue, 11 Jun 2013 03:43:56 +0000
Final status: Tue, 11 Jun 2013 03:43:56 +0000  Finished successfully:2

mid$ more row?.out
::::::::::::::
row1.out
::::::::::::::
params: 86 99 files: f1.dat f2.dat
::::::::::::::
row2.out
::::::::::::::
params: 87 98 files: f3.dat f4.dat
mid$ 


----- Original Message -----
> From: "TJ Lane" <tjlane at stanford.edu>
> To: swift-user at ci.uchicago.edu
> Sent: Monday, June 10, 2013 8:15:17 PM
> Subject: [Swift-user] Advice on mapping input arguments
> 
> 
> 
> 
> Swift Users,
> 
> I am wondering if I could get some advice on the best way to do the
> following in Swift: I want to run a series of simulations performing
> a parameter scan, for each parameter combination farming the work
> out to clusters I have access to here at Stanford, and collect the
> results back onto my desktop.
> 
> I've gotten some minimal working examples of swift up and running,
> but hit a roadblock on something quite simple: what's the best way
> to pass a large number of parameters into a swift script? I have a
> big list of parameter combinations I'd like to run, and am searching
> for a sane way to pass all of these into my swift app call.
> 
> Originally, I thought I'd be able to use the CSV mapper to pass a
> bunch of arguments from a CSV file into swift -- it seemed perfect!
> As a bonus, I hoped the CSV file would act as a record of my work,
> namely what parameters were used to generate what file. But it seems
> that the CSV mapper automatically maps the entries in the CSV file
> to swift "mapper" objects -- i.e., it expects my CSV data fields are
> all files, where as I want some to be ints or floats that get passed
> directly to the arguments of my command-line script on the slave
> machine(s).
> 
> 
> For concreteness, here is a test CSV I was working with:
> 
> 
> coords,qvals,numphi,numshots,nummolec,photons,parallel,output_filename
> gold_5nm.coor,gold_qvals.txt,3600,10,1,0.25,12,out0.ring
> gold_5nm.coor,gold_qvals.txt,3600,10,2,0.5,12,out1.ring
> gold_5nm.coor,gold_qvals.txt,3600,10,4,1.0,12,out2.ring
> gold_5nm.coor,gold_qvals.txt,3600,10,8,2.0,12,out3.ring
> gold_5nm.coor,gold_qvals.txt,3600,10,16,4.0,12,out4.ring
> 
> 
> and my (non-functional) swift script, which will show what I was
> trying to do:
> 
> # shoot.swift
> 
> type messagefile;
> type pdbfile;
> type shotfile;
> 
> type shootargs{
> pdbfile coords;
> messagefile qvals;
> int numphi;
> int numshots;
> int nummolec;
> int photons;
> int parallel;
> string output_filename;
> }
> 
> app (shotfile outputfile) shootsim (shootargs args) {
> polshoot "-s" @args.coords "-f" @args.qvals "-x" args.numphi "-n"
> args.numshots "-m" args.nummolec "-g" args.photons "-p"
> args.parallel "-o" @outputfile;
> }
> 
> 
> shootargs myargs[] <csv_mapper;file="particles_per_shot.csv">;
> 
> foreach a in myargs {
> shotfile o; // this could be something like myargs.output_filename
> o = shootsim(a);
> }
> 
> 
> I'm wondering if someone who's worked a bit with swift can give me a
> recommendation on how to proceed. Right now I'm playing with just
> writing a huge number of flat text files, each one containing the
> parameter flags that will then get cat'd into the arguments of my
> command-line script "polshoot" on the slave end. This is inelegant
> for obvious reasons, since I'll have a huge number of input files
> and no easy way to keep track of which input matches what output...
> 
> 
> If anyone has advice, I'm all ears!
> 
> 
> Thanks,
> 
> TJ
> 
> 
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user