<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Mike,<br><br>Awesome, thanks for your help! Seems to be working like a charm.<br><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">
TJ<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 10, 2013 at 9:26 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">TJ,<br>
<br>
For the specific example you posted, the technique shown in the prior message works well.<br>
<br>
Here's shoot.swift converted to use that approach.<br>
<br>
- Mike<br>
<br>
mid$ cat shoot.swift<br>
<br>
type qvalfile;<br>
type coordfile;<br>
type shotfile;<br>
<br>
type shootargs{<br>
string coords;<br>
string qvals;<br>
int numphi;<br>
int numshots;<br>
int nummolec;<br>
float photons;<br>
int parallel;<br>
string output_filename;<br>
}<br>
<br>
app (shotfile outfile) shootsim (shootargs args, coordfile coords, qvalfile qvals) {<br>
echo "polshoot" "-s" @coords "-f" @qvals "-x" args.numphi "-n" args.numshots<br>
"-m" args.nummolec "-g" args.photons "-p" args.parallel "-o" @outfile<br>
stdout=@outfile; # Note double use of outfile, just for testing/demo<br>
}<br>
<br>
shootargs myargs[] = readData("particles.dat");<br>
<br>
foreach a in myargs {<br>
shotfile output <single_file_mapper; file=a.output_filename>;<br>
coordfile coords <single_file_mapper; file=a.coords>;<br>
qvalfile qvals <single_file_mapper; file=a.qvals>;<br>
output = shootsim(a, coords, qvals);<br>
}<br>
<br>
mid$ cat particles.dat<br>
coords qvals numphi numshots nummolec photons parallel output_filename<br>
gold_5nm.coor gold_qvals.txt 3600 10 1 0.25 12 out0.ring<br>
gold_5nm.coor gold_qvals.txt 3600 10 2 0.5 12 out1.ring<br>
gold_5nm.coor gold_qvals.txt 3600 10 4 1.0 12 out2.ring<br>
gold_5nm.coor gold_qvals.txt 3600 10 8 2.0 12 out3.ring<br>
gold_5nm.coor gold_qvals.txt 3600 10 16 4.0 12 out4.ring<br>
<br>
<br>
mid$ swift shoot.swift<br>
Swift 0.94 swift-r6414 (swift modified locally) cog-r3648<br>
<br>
RunID: 20130611-0423-gfp7pohe<br>
Progress: time: Tue, 11 Jun 2013 04:23:18 +0000<br>
Progress: time: Tue, 11 Jun 2013 04:23:19 +0000 Stage in:1 Finished successfully:4<br>
Final status: Tue, 11 Jun 2013 04:23:19 +0000 Finished successfully:5<br>
<br>
mid$ more out?.ring<br>
::::::::::::::<br>
out0.ring<br>
::::::::::::::<br>
polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 1 -g 0.25 -p 12 -o out0.ring<br>
::::::::::::::<br>
out1.ring<br>
::::::::::::::<br>
polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 2 -g 0.5 -p 12 -o out1.ring<br>
::::::::::::::<br>
out2.ring<br>
::::::::::::::<br>
polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 4 -g 1 -p 12 -o out2.ring<br>
::::::::::::::<br>
out3.ring<br>
::::::::::::::<br>
polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 8 -g 2 -p 12 -o out3.ring<br>
::::::::::::::<br>
out4.ring<br>
::::::::::::::<br>
polshoot -s gold_5nm.coor -f gold_qvals.txt -x 3600 -n 10 -m 16 -g 4 -p 12 -o out4.ring<br>
mid$<br>
<br>
<br>
<br>
----- Original Message -----<br>
> From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> To: "TJ Lane" <<a href="mailto:tjlane@stanford.edu">tjlane@stanford.edu</a>><br>
> Cc: <a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a><br>
> Sent: Monday, June 10, 2013 10:58:55 PM<br>
> Subject: Re: [Swift-user] Advice on mapping input arguments<br>
><br>
> Hi TJ,<br>
><br>
> Here's a quick initial thought. There might be better ways to do<br>
> this.<br>
><br>
> The problem as you mention is that there is currently no mapper or<br>
> function that lets you easily handle a mixture of scalars and files<br>
> in a struct. I think that readData and/or mappers should/could<br>
> handle this, but Ive been outvoted in past discussions of this.<br>
><br>
> So for now, assuming you have more scalar params than file params,<br>
> you can just leave the filenames as strings in the parameter struct,<br>
> then map the smaller number of files from these strings, and call<br>
> your app() using the param struct to pass all the scalar params, and<br>
> pass the mapped files as additional input and output parameters.<br>
><br>
> This retains, I think, the benefit of having a parameter file that<br>
> provides a handy record of the run's parameters.<br>
><br>
> Here's an initial example that I *think* is in the spirit of the<br>
> longer one you supplied below.<br>
><br>
> If this sounds like a reasonable approach I think we can use the same<br>
> technique to run your example.<br>
><br>
> I provide I think all the files you need to test this.<br>
><br>
> I'll look more carefully at your example now to see if this is indeed<br>
> what you're trying to do.<br>
><br>
> - Mike<br>
><br>
> mid$ cat params.txt<br>
> p1 p2 fn1 fn2 ofn<br>
> 86 99 f1.dat f2.dat row1.out<br>
> 87 98 f3.dat f4.dat row2.out<br>
><br>
> mid$ cat structs.swift<br>
><br>
> type file;<br>
><br>
> type params {<br>
> int p1;<br>
> int p2;<br>
> string fn1;<br>
> string fn2;<br>
> string ofn;<br>
> };<br>
><br>
> app (file out) echo (params p, file f1, file f2)<br>
> {<br>
> echo stdout=@out "params:" p.p1 p.p2 "files:" @f1 @f2;<br>
> }<br>
><br>
> params plist[] = readData("params.txt");<br>
><br>
> foreach p, i in plist {<br>
> file f1 <single_file_mapper; file=p.fn1>;<br>
> file f2 <single_file_mapper; file=p.fn2>;<br>
> file of <single_file_mapper; file=p.ofn>;<br>
> of = echo(p,f1,f2);<br>
> }<br>
><br>
> mid$ cat ~/.swift/swift.properties<br>
><br>
> sites.file=sites.xml<br>
> tc.file=apps<br>
><br>
> status.mode=provider<br>
> use.provider.staging=false<br>
> use.wrapper.staging=false<br>
> wrapperlog.always.transfer=true<br>
> execution.retries=0<br>
> lazy.errors=false<br>
> provider.staging.pin.swiftfiles=false<br>
> sitedir.keep=true<br>
> file.gc.enabled=false<br>
> #tcp.port.range=50000,51000<br>
><br>
> mid$ cat apps<br>
> localhost echo echo<br>
><br>
> mid$ cat sites.xml<br>
> <config><br>
> <pool handle="localhost"><br>
> <execution provider="local"/><br>
> <filesystem provider="local"/><br>
> <workdirectory>/scratch/midway/{env.USER}/swiftwork</workdirectory><br>
> </pool><br>
> </config><br>
><br>
> mid$ ls -l f?.dat<br>
> -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f1.dat<br>
> -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f2.dat<br>
> -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f3.dat<br>
> -rw-rw-r-- 1 wilde wilde 4 Jun 10 22:40 f4.dat<br>
><br>
> mid$ swift structs.swift<br>
> Swift 0.94 swift-r6414 (swift modified locally) cog-r3648<br>
><br>
> RunID: 20130611-0343-so454ho1<br>
> Progress: time: Tue, 11 Jun 2013 03:43:56 +0000<br>
> Final status: Tue, 11 Jun 2013 03:43:56 +0000 Finished<br>
> successfully:2<br>
><br>
> mid$ more row?.out<br>
> ::::::::::::::<br>
> row1.out<br>
> ::::::::::::::<br>
> params: 86 99 files: f1.dat f2.dat<br>
> ::::::::::::::<br>
> row2.out<br>
> ::::::::::::::<br>
> params: 87 98 files: f3.dat f4.dat<br>
> mid$<br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "TJ Lane" <<a href="mailto:tjlane@stanford.edu">tjlane@stanford.edu</a>><br>
> > To: <a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a><br>
> > Sent: Monday, June 10, 2013 8:15:17 PM<br>
> > Subject: [Swift-user] Advice on mapping input arguments<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Swift Users,<br>
> ><br>
> > I am wondering if I could get some advice on the best way to do the<br>
> > following in Swift: I want to run a series of simulations<br>
> > performing<br>
> > a parameter scan, for each parameter combination farming the work<br>
> > out to clusters I have access to here at Stanford, and collect the<br>
> > results back onto my desktop.<br>
> ><br>
> > I've gotten some minimal working examples of swift up and running,<br>
> > but hit a roadblock on something quite simple: what's the best way<br>
> > to pass a large number of parameters into a swift script? I have a<br>
> > big list of parameter combinations I'd like to run, and am<br>
> > searching<br>
> > for a sane way to pass all of these into my swift app call.<br>
> ><br>
> > Originally, I thought I'd be able to use the CSV mapper to pass a<br>
> > bunch of arguments from a CSV file into swift -- it seemed perfect!<br>
> > As a bonus, I hoped the CSV file would act as a record of my work,<br>
> > namely what parameters were used to generate what file. But it<br>
> > seems<br>
> > that the CSV mapper automatically maps the entries in the CSV file<br>
> > to swift "mapper" objects -- i.e., it expects my CSV data fields<br>
> > are<br>
> > all files, where as I want some to be ints or floats that get<br>
> > passed<br>
> > directly to the arguments of my command-line script on the slave<br>
> > machine(s).<br>
> ><br>
> ><br>
> > For concreteness, here is a test CSV I was working with:<br>
> ><br>
> ><br>
> > coords,qvals,numphi,numshots,nummolec,photons,parallel,output_filename<br>
> > gold_5nm.coor,gold_qvals.txt,3600,10,1,0.25,12,out0.ring<br>
> > gold_5nm.coor,gold_qvals.txt,3600,10,2,0.5,12,out1.ring<br>
> > gold_5nm.coor,gold_qvals.txt,3600,10,4,1.0,12,out2.ring<br>
> > gold_5nm.coor,gold_qvals.txt,3600,10,8,2.0,12,out3.ring<br>
> > gold_5nm.coor,gold_qvals.txt,3600,10,16,4.0,12,out4.ring<br>
> ><br>
> ><br>
> > and my (non-functional) swift script, which will show what I was<br>
> > trying to do:<br>
> ><br>
> > # shoot.swift<br>
> ><br>
> > type messagefile;<br>
> > type pdbfile;<br>
> > type shotfile;<br>
> ><br>
> > type shootargs{<br>
> > pdbfile coords;<br>
> > messagefile qvals;<br>
> > int numphi;<br>
> > int numshots;<br>
> > int nummolec;<br>
> > int photons;<br>
> > int parallel;<br>
> > string output_filename;<br>
> > }<br>
> ><br>
> > app (shotfile outputfile) shootsim (shootargs args) {<br>
> > polshoot "-s" @args.coords "-f" @args.qvals "-x" args.numphi "-n"<br>
> > args.numshots "-m" args.nummolec "-g" args.photons "-p"<br>
> > args.parallel "-o" @outputfile;<br>
> > }<br>
> ><br>
> ><br>
> > shootargs myargs[] <csv_mapper;file="particles_per_shot.csv">;<br>
> ><br>
> > foreach a in myargs {<br>
> > shotfile o; // this could be something like myargs.output_filename<br>
> > o = shootsim(a);<br>
> > }<br>
> ><br>
> ><br>
> > I'm wondering if someone who's worked a bit with swift can give me<br>
> > a<br>
> > recommendation on how to proceed. Right now I'm playing with just<br>
> > writing a huge number of flat text files, each one containing the<br>
> > parameter flags that will then get cat'd into the arguments of my<br>
> > command-line script "polshoot" on the slave end. This is inelegant<br>
> > for obvious reasons, since I'll have a huge number of input files<br>
> > and no easy way to keep track of which input matches what output...<br>
> ><br>
> ><br>
> > If anyone has advice, I'm all ears!<br>
> ><br>
> ><br>
> > Thanks,<br>
> ><br>
> > TJ<br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Swift-user mailing list<br>
> > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> _______________________________________________<br>
> Swift-user mailing list<br>
> <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
><br>
</blockquote></div><br></div>