<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Mike,<br><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Is there support for other schedulers? Specifically I have a cluster running LSF I'd like to also farm jobs out to. Maybe this is documented somewhere? Haven't been able to locate it yet.<br>
<br>Thanks a lot for your help! Looking forward to playing more w/Swift.<br><br>TJ<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 3, 2013 at 8:38 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov" target="_blank">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I forgot to also mention: the example below with the "ssh-cl" ("ssh command line") provider also assumes that you can do a password-less ssh command from your workstation to your PBS head node. Ie, that you have ssh keys in place on the head node and that youre using an ssh agent.<br>
<br>
The standard Swift ssh provider (eg using provider=coaster jobmanager=ssh:pbs) uses a file called $HOME/.ssh/auth.defaults to specify ssh passwords or passphrases, or for better security swift will prompt for these.<br>
<br>
We tend to use and recommend the newer ssh-cl for both security and convenience.<br>
<br>
- Mike<br>
<br>
<br>
----- Original Message -----<br>
> From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> To: "Robert McGibbon" <<a href="mailto:rmcgibbo@gmail.com">rmcgibbo@gmail.com</a>><br>
> Cc: <a href="mailto:swift-user@ci.uchicago.edu">swift-user@ci.uchicago.edu</a><br>
> Sent: Monday, June 3, 2013 10:27:45 PM<br>
> Subject: Re: [Swift-user] Setting up Swift at Stanford<br>
><br>
> Hi Robert,<br>
><br>
> To run swift from a workstation that can ssh to one or more cluster<br>
> head nodes, use a sites file like this:<br>
><br>
> <pool handle="vsp-compute"><br>
> <execution provider="coaster" jobmanager="ssh-cl:pbs"<br>
> url="<a href="http://vsp-compute-01.stanford.edu" target="_blank">vsp-compute-01.stanford.edu</a>"/><br>
> <profile namespace="globus" key="jobsPerNode">1</profile><br>
> <profile namespace="globus" key="lowOverAllocation">100</profile><br>
> <profile namespace="globus"<br>
> key="highOverAllocation">100</profile><br>
> <profile namespace="globus" key="maxtime">3600</profile><br>
> <profile namespace="globus" key="maxWalltime">00:05:00</profile><br>
> <profile namespace="globus" key="queue">default</profile><br>
> <profile namespace="globus" key="slots">5</profile><br>
> <profile namespace="globus" key="maxnodes">1</profile><br>
> <profile namespace="globus" key="nodeGranularity">1</profile><br>
> <profile namespace="karajan" key="jobThrottle">1.00</profile><br>
> <profile namespace="karajan" key="initialScore">10000</profile><br>
> <workdirectory>/scratch/rmcgibbo/swiftwork</workdirectory><br>
> </pool><br>
><br>
> This specifies that Swift should:<br>
><br>
> - use the "coaster" provider, which enables Swift to ssh to another<br>
> system and qsub from there:<br>
><br>
> <execution provider="coaster" jobmanager="ssh-cl:pbs"<br>
> url="<a href="http://vsp-compute-01.stanford.edu" target="_blank">vsp-compute-01.stanford.edu</a>"/><br>
><br>
> - run up to 100 Swift app() tasks in parallel on the remote system:<br>
><br>
> <profile namespace="karajan" key="jobThrottle">1.00</profile><br>
> <profile namespace="karajan" key="initialScore">10000</profile><br>
><br>
> - app() tasks should be limited to 5 minutes walltime:<br>
><br>
> <profile namespace="globus" key="maxWalltime">00:05:00</profile><br>
><br>
> - app() tasks will be run within PBS coaster "pilot" jobs. Each PBS<br>
> job should have a walltime of 750 seconds:<br>
><br>
> <profile namespace="globus" key="lowOverAllocation">100</profile><br>
> <profile namespace="globus" key="highOverAllocation">100</profile><br>
> <profile namespace="globus" key="maxtime">750</profile><br>
><br>
> - Up to 5 concurrent PBS coaster jobs each asking for 1 node will be<br>
> submitted to the default queue:<br>
><br>
> <profile namespace="globus" key="queue">default</profile><br>
> <profile namespace="globus" key="slots">5</profile><br>
> <profile namespace="globus" key="maxnodes">1</profile><br>
> <profile namespace="globus" key="nodeGranularity">1</profile><br>
><br>
> - Swift should run only one app() task at a time within each PBS job<br>
> slot:<br>
><br>
> <profile namespace="globus" key="jobsPerNode">1</profile><br>
><br>
> - On the remote PBS cluster, create per-run directories under this<br>
> work directory:<br>
><br>
> <workdirectory>/scratch/rmcgibbo/swiftwork</workdirectory><br>
><br>
> - And stage data to the site by using local copy operations:<br>
><br>
> <filesystem provider="local"/><br>
><br>
> You can make the sites.xml entry more user-independent using, e.g.:<br>
><br>
> <workdirectory>/scratch/{env.USER}/swiftwork</workdirectory><br>
><br>
> The overall sites entry above assumes:<br>
><br>
> - That /scratch/rmcgibbo is mounted on both the Swift run host and on<br>
> the remote PBS system.<br>
><br>
> If there is no common shared filesystem, Swift can use a data<br>
> transport technique called "coaster provider staging" to move the<br>
> data for you. This is specified in the swift.properties file.<br>
><br>
> In many cases, with a shared filesystem bewteen the Swift client host<br>
> and the execution cluster, its desirable to turn off staging<br>
> altogether. This is done using a mode called "direct" data<br>
> management (see<br>
> <a href="http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_collective_data_management" target="_blank">http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_collective_data_management</a>.<br>
> This is being simplified for future releases.)<br>
><br>
> - That each PBS job is given one CPU core, not one full node.<br>
><br>
> The PBS ppn attribute can be specified to request a specific number<br>
> of cores (processors) per node:<br>
><br>
> <profile namespace="globus" key="ppn">16</profile><br>
><br>
> ...and then that each coaster pilot job should run up to 16 Swift<br>
> app() tasks at once:<br>
><br>
> <profile namespace="globus" key="jobsPerNode">16</profile><br>
><br>
> For more info on coasters, see:<br>
> <a href="http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_coasters" target="_blank">http://www.ci.uchicago.edu/swift/guides/trunk/userguide/userguide.html#_coasters</a><br>
> and: <a href="http://www.ci.uchicago.edu/swift/papers/UCC-coasters.pdf" target="_blank">http://www.ci.uchicago.edu/swift/papers/UCC-coasters.pdf</a><br>
><br>
> For more examples on site configurations, see:<br>
><br>
> <a href="http://www.ci.uchicago.edu/swift/guides/trunk/siteguide/siteguide.html" target="_blank">http://www.ci.uchicago.edu/swift/guides/trunk/siteguide/siteguide.html</a><br>
><br>
> And lastly, note that in your initial sites.xml below:<br>
><br>
> - Omitting the filesystem provider tag is typically only done when<br>
> "use.provider.staging" is specified in the swift.properties config<br>
> file<br>
><br>
> - The stagingMethod tag only applies to provider staging.<br>
><br>
> We're working hard to document all this better and provider a better<br>
> set of illustrated examples and templates for common site<br>
> configurations. In the meantime, we'll help you create a set of<br>
> useful configurations for your site(s).<br>
><br>
> Regards,<br>
><br>
> - Mike<br>
><br>
> > We just heard about the swift project from some colleagues at U<br>
> > Chicago, and we're interested in trying it out with some of our<br>
> > compute resources at Stanford to run parallel molecular dynamics<br>
> > and<br>
> > x-ray scatting simulations. Currently, I'm most interested in<br>
> > setting up the environment such that I can submit my swift script<br>
> > on<br>
> > a local workstation, with execution on a few different clusters.<br>
> > The<br>
> > head nodes of our local clusters are accessible via ssh, and then<br>
> > job execution is scheduled with pbs.<br>
> ><br>
> > When I run swift, it can't seem to find qsub on the cluster.<br>
> ><br>
> > rmcgibbo@Roberts-MacBook-Pro-2 ~/projects/swift<br>
> > $ swift -sites.file sites.xml hello.swift -tc.file tc.data<br>
> > Swift 0.94 swift-r6492 cog-r3658<br>
> ><br>
> > RunID: 20130603-1704-5xii8svc<br>
> > Progress: time: Mon, 03 Jun 2013 17:04:10 -0700<br>
> > 2013-06-03 17:04:10.735 java[77051:1f07] Loading Maximizer into<br>
> > bundle: com.apple.javajdk16.cmd<br>
> > 2013-06-03 17:04:11.410 java[77051:1f07] Maximizer: Unsupported<br>
> > window created of class: CocoaAppWindow<br>
> > Progress: time: Mon, 03 Jun 2013 17:04:13 -0700 Stage in:1<br>
> > Execution failed:<br>
> > Exception in uname:<br>
> > Arguments: [-a]<br>
> > Host: vsp-compute<br>
> > Directory: hello-20130603-1704-5xii8svc/jobs/y/uname-ydyn5fal<br>
> > Caused by:<br>
> > Cannot submit job: Cannot run program "qsub": error=2, No such file<br>
> > or directory<br>
> > uname, hello.swift, line 8<br>
> ><br>
> > When I switch the execution provider from pbs to ssh, the hob runs<br>
> > successfully, but only on the head node of the vsp-compute cluster.<br>
> > I'd like to run instead using the cluster's pbs queue. Any help<br>
> > would be greatly appreciated.<br>
> ><br>
> > -Robert<br>
> > Graduate Student, Pande Lab<br>
> > Stanford University, Department of Chemistry<br>
> ><br>
> > p.s.<br>
> ><br>
> > My sitess.xml file is<br>
> > ```<br>
> > <config><br>
> > <pool handle="vsp-compute"><br>
> > <filesystem provider="ssh" url=" <a href="http://vsp-compute-01.stanford.edu" target="_blank">vsp-compute-01.stanford.edu</a> "/><br>
> > <execution provider="pbs" jobmanager="ssh:pbs" url="<br>
> > <a href="http://vsp-compute-01.stanford.edu" target="_blank">vsp-compute-01.stanford.edu</a> "/><br>
> ><br>
> > <profile namespace="globus" key="maxtime">750</profile><br>
> > <profile namespace="globus" key="jobsPerNode">1</profile><br>
> > <profile namespace="globus" key="queue">default</profile><br>
> > <profile namespace="swift" key="stagingMethod">file</profile><br>
> ><br>
> > <workdirectory>/scratch/rmcgibbo/swiftwork</workdirectory><br>
> > </pool><br>
> ><br>
> > <!-- End --><br>
> > </config><br>
> > ```<br>
> ><br>
> > My SwiftScript is<br>
> > ```<br>
> > #hello.swift<br>
> > type file;<br>
> ><br>
> > app (file o) uname() {<br>
> > uname "-a" stdout=@o;<br>
> > }<br>
> > file outfile <"uname.txt">;<br>
> ><br>
> > outfile = uname();<br>
> > ```<br>
> > _______________________________________________<br>
> > Swift-user mailing list<br>
> > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> _______________________________________________<br>
> Swift-user mailing list<br>
> <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
><br>
_______________________________________________<br>
Swift-user mailing list<br>
<a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
</blockquote></div><br></div>