[Swift-user] Advice on mapping input arguments
TJ Lane
tjlane at stanford.edu
Tue Jun 11 13:50:18 CDT 2013
Mike,
Awesome, thanks for your help! Seems to be working like a charm.
TJ
On Mon, Jun 10, 2013 at 9:26 PM, Michael Wilde <wilde at mcs.anl.gov> wrote:
> TJ,
>
> For the specific example you posted, the technique shown in the prior
> message works well.
>
> Here's shoot.swift converted to use that approach.
>
> - Mike
>
> mid$ cat shoot.swift
>
> type qvalfile;
> type coordfile;
> type shotfile;
>
> type shootargs{
> string coords;
> string qvals;
> int numphi;
> int numshots;
> int nummolec;
> float photons;
> int parallel;
> string output_filename;
> }
>
> app (shotfile outfile) shootsim (shootargs args, coordfile coords,
> qvalfile qvals) {
> echo "polshoot" "-s" @coords "-f" @qvals "-x" args.numphi "-n"
> args.numshots
> "-m" args.nummolec "-g" args.photons "-p"
> args.parallel "-o" @outfile
> stdout=@outfile; # Note double use of outfile, just
> for testing/demo
> }
>
> shootargs myargs[] = readData("particles.dat");
>
> foreach a in myargs {
> shotfile output <single_file_mapper; file=a.output_filename>;
> coordfile coords <single_file_mapper; file=a.coords>;
> qvalfile qvals <single_file_mapper; file=a.qvals>;
> output = shootsim(a, coords, qvals);
> }
>
> mid$ cat particles.dat
> coords qvals numphi numshots nummolec photons parallel
> output_filename
> gold_5nm.coor gold_qvals.txt 3600 10 1 0.25 12
> out0.ring
> gold_5nm.coor gold_qvals.txt 3600 10 2 0.5 12
> out1.ring
> gold_5nm.coor gold_qvals.txt 3600 10 4 1.0 12
> out2.ring
> gold_5nm.coor gold_qvals.txt 3600 10 8 2.0 12
> out3.ring
> gold_5nm.coor gold_qvals.txt 3600 10 16 4.0 12
> out4.ring
>
>
> mid$ swift shoot.swift
> Swift 0.94 swift-r6414 (swift modified locally) cog-r3648
>
> RunID: 20130611-0423-gfp7pohe
> Progress: time: Tue, 11 Jun 2013 04:23:18 +0000
> Progress: time: Tue, 11 Jun 2013 04:23:19 +0000 Stage in:1 Finished
> successfully:4
> Final status: Tue, 11 Jun 2013 04:23:19 +0000 Finished successfully:5
>
> mid$ more out?.ring
> ::::::::::::::
> out0.ring
> ::::::::::::::
> polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 1 -g 0.25 -p
> 12 -o out0.ring
> ::::::::::::::
> out1.ring
> ::::::::::::::
> polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 2 -g 0.5 -p
> 12 -o out1.ring
> ::::::::::::::
> out2.ring
> ::::::::::::::
> polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 4 -g 1 -p 12
> -o out2.ring
> ::::::::::::::
> out3.ring
> ::::::::::::::
> polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 8 -g 2 -p 12
> -o out3.ring
> ::::::::::::::
> out4.ring
> ::::::::::::::
> polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 16 -g 4 -p 12
> -o out4.ring
> mid$
>
>
>
> ----- Original Message -----
> > From: "Michael Wilde" <wilde at mcs.anl.gov>
> > To: "TJ Lane" <tjlane at stanford.edu>
> > Cc: swift-user at ci.uchicago.edu
> > Sent: Monday, June 10, 2013 10:58:55 PM
> > Subject: Re: [Swift-user] Advice on mapping input arguments
> >
> > Hi TJ,
> >
> > Here's a quick initial thought. There might be better ways to do
> > this.
> >
> > The problem as you mention is that there is currently no mapper or
> > function that lets you easily handle a mixture of scalars and files
> > in a struct. I think that readData and/or mappers should/could
> > handle this, but Ive been outvoted in past discussions of this.
> >
> > So for now, assuming you have more scalar params than file params,
> > you can just leave the filenames as strings in the parameter struct,
> > then map the smaller number of files from these strings, and call
> > your app() using the param struct to pass all the scalar params, and
> > pass the mapped files as additional input and output parameters.
> >
> > This retains, I think, the benefit of having a parameter file that
> > provides a handy record of the run's parameters.
> >
> > Here's an initial example that I *think* is in the spirit of the
> > longer one you supplied below.
> >
> > If this sounds like a reasonable approach I think we can use the same
> > technique to run your example.
> >
> > I provide I think all the files you need to test this.
> >
> > I'll look more carefully at your example now to see if this is indeed
> > what you're trying to do.
> >
> > - Mike
> >
> > mid$ cat params.txt
> > p1 p2 fn1 fn2 ofn
> > 86 99 f1.dat f2.dat row1.out
> > 87 98 f3.dat f4.dat row2.out
> >
> > mid$ cat structs.swift
> >
> > type file;
> >
> > type params {
> > int p1;
> > int p2;
> > string fn1;
> > string fn2;
> > string ofn;
> > };
> >
> > app (file out) echo (params p, file f1, file f2)
> > {
> > echo stdout=@out "params:" p.p1 p.p2 "files:" @f1 @f2;
> > }
> >
> > params plist[] = readData("params.txt");
> >
> > foreach p, i in plist {
> > file f1 <single_file_mapper; file=p.fn1>;
> > file f2 <single_file_mapper; file=p.fn2>;
> > file of <single_file_mapper; file=p.ofn>;
> > of = echo(p,f1,f2);
> > }
> >
> > mid$ cat ~/.swift/swift.properties
> >
> > sites.file=sites.xml
> > tc.file=apps
> >
> > status.mode=provider
> > use.provider.staging=false
> > use.wrapper.staging=false
> > wrapperlog.always.transfer=true
> > execution.retries=0
> > lazy.errors=false
> > provider.staging.pin.swiftfiles=false
> > sitedir.keep=true
> > file.gc.enabled=false
> > #tcp.port.range=50000,51000
> >
> > mid$ cat apps
> > localhost echo echo
> >
> > mid$ cat sites.xml
> > <config>
> > <pool handle="localhost">
> > <execution provider="local"/>
> > <filesystem provider="local"/>
> > <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory>
> > </pool>
> > </config>
> >
> > mid$ ls -l f?.dat
> > -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f1.dat
> > -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f2.dat
> > -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f3.dat
> > -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f4.dat
> >
> > mid$ swift structs.swift
> > Swift 0.94 swift-r6414 (swift modified locally) cog-r3648
> >
> > RunID: 20130611-0343-so454ho1
> > Progress: time: Tue, 11 Jun 2013 03:43:56 +0000
> > Final status: Tue, 11 Jun 2013 03:43:56 +0000 Finished
> > successfully:2
> >
> > mid$ more row?.out
> > ::::::::::::::
> > row1.out
> > ::::::::::::::
> > params: 86 99 files: f1.dat f2.dat
> > ::::::::::::::
> > row2.out
> > ::::::::::::::
> > params: 87 98 files: f3.dat f4.dat
> > mid$
> >
> >
> > ----- Original Message -----
> > > From: "TJ Lane" <tjlane at stanford.edu>
> > > To: swift-user at ci.uchicago.edu
> > > Sent: Monday, June 10, 2013 8:15:17 PM
> > > Subject: [Swift-user] Advice on mapping input arguments
> > >
> > >
> > >
> > >
> > > Swift Users,
> > >
> > > I am wondering if I could get some advice on the best way to do the
> > > following in Swift: I want to run a series of simulations
> > > performing
> > > a parameter scan, for each parameter combination farming the work
> > > out to clusters I have access to here at Stanford, and collect the
> > > results back onto my desktop.
> > >
> > > I've gotten some minimal working examples of swift up and running,
> > > but hit a roadblock on something quite simple: what's the best way
> > > to pass a large number of parameters into a swift script? I have a
> > > big list of parameter combinations I'd like to run, and am
> > > searching
> > > for a sane way to pass all of these into my swift app call.
> > >
> > > Originally, I thought I'd be able to use the CSV mapper to pass a
> > > bunch of arguments from a CSV file into swift -- it seemed perfect!
> > > As a bonus, I hoped the CSV file would act as a record of my work,
> > > namely what parameters were used to generate what file. But it
> > > seems
> > > that the CSV mapper automatically maps the entries in the CSV file
> > > to swift "mapper" objects -- i.e., it expects my CSV data fields
> > > are
> > > all files, where as I want some to be ints or floats that get
> > > passed
> > > directly to the arguments of my command-line script on the slave
> > > machine(s).
> > >
> > >
> > > For concreteness, here is a test CSV I was working with:
> > >
> > >
> > > coords,qvals,numphi,numshots,nummolec,photons,parallel,output_filename
> > > gold_5nm.coor,gold_qvals.txt,3600,10,1,0.25,12,out0.ring
> > > gold_5nm.coor,gold_qvals.txt,3600,10,2,0.5,12,out1.ring
> > > gold_5nm.coor,gold_qvals.txt,3600,10,4,1.0,12,out2.ring
> > > gold_5nm.coor,gold_qvals.txt,3600,10,8,2.0,12,out3.ring
> > > gold_5nm.coor,gold_qvals.txt,3600,10,16,4.0,12,out4.ring
> > >
> > >
> > > and my (non-functional) swift script, which will show what I was
> > > trying to do:
> > >
> > > # shoot.swift
> > >
> > > type messagefile;
> > > type pdbfile;
> > > type shotfile;
> > >
> > > type shootargs{
> > > pdbfile coords;
> > > messagefile qvals;
> > > int numphi;
> > > int numshots;
> > > int nummolec;
> > > int photons;
> > > int parallel;
> > > string output_filename;
> > > }
> > >
> > > app (shotfile outputfile) shootsim (shootargs args) {
> > > polshoot "-s" @args.coords "-f" @args.qvals "-x" args.numphi "-n"
> > > args.numshots "-m" args.nummolec "-g" args.photons "-p"
> > > args.parallel "-o" @outputfile;
> > > }
> > >
> > >
> > > shootargs myargs[] <csv_mapper;file="particles_per_shot.csv">;
> > >
> > > foreach a in myargs {
> > > shotfile o; // this could be something like myargs.output_filename
> > > o = shootsim(a);
> > > }
> > >
> > >
> > > I'm wondering if someone who's worked a bit with swift can give me
> > > a
> > > recommendation on how to proceed. Right now I'm playing with just
> > > writing a huge number of flat text files, each one containing the
> > > parameter flags that will then get cat'd into the arguments of my
> > > command-line script "polshoot" on the slave end. This is inelegant
> > > for obvious reasons, since I'll have a huge number of input files
> > > and no easy way to keep track of which input matches what output...
> > >
> > >
> > > If anyone has advice, I'm all ears!
> > >
> > >
> > > Thanks,
> > >
> > > TJ
> > >
> > >
> > >
> > > _______________________________________________
> > > Swift-user mailing list
> > > Swift-user at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20130611/9472af8f/attachment.html>
More information about the Swift-user
mailing list