[Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?

Mon Oct 17 23:40:57 CDT 2011

Justin, I have a few questions about this page:

"If using Coasters, the count attribute is the number of running worker.pl scripts (1); this is the number of nodes to use."

What is the "count" attribute?

"The ppn attribute is available but should be left to default to 1."

This is because we are running one program per node, worker.pl, and giving it (typically) the full "depth" of the node (eg 24 cores) to manage?

  <profile namespace="globus" key="providerAttributes">
    pbs.aprun;pbs.mpp;depth=24
  </profile>

I find this attribute confusing. I assume this means:

pbs.aprun: generate an aprun command for Crays in the pbs submit file.
pbs.mpp: generate Cray-style mppwidth and mppnppn attributes?  Of what value? Can a value other than 1 be specified?
depth= : why does depth not have a pbs.depth prefix?
ppn: how would that be specified?  Leave off ppn only for Cray?
What other PBS attributes can be specified, and how are they processed?

On the PBS page above the Cray page you say:

<profile namespace="globus" key=" ... "> ... </profile>
pbs.properties: Adds the value to the end of the "#PBS -l " line.  See Fusion for an example use case
pbs.mpp: If value is set, use mppwidth/mppnppn instead of nodes/ppn in PBS submit file
pbs.aprun: If value is set, use aprun-based command line 
pbs.resources: Adds the value to a new "#PBS -l" line.  

Is the key always providerAttributes?
pbs.mpp by "if value is set" you mean "if this string is present in the tags value, separated from other strings by semicolons"?

pbs.properties and pbs.resources - are the same except for where they are inserted?  Are these followed by a pbs -l attributes such as "nodes=10:ppn=4"? 

I think several examples are needed to be able to understand how to use (and document) these.

Could you elaborate a bit in the swiftdevel page?  Or are these documented already elsewhere?

Thanks,

- Mike

----- Original Message -----
> From: "Justin M Wozniak" <wozniak at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
> Sent: Monday, October 17, 2011 10:02:15 AM
> Subject: Re: [Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?
> My notes about Beagle are at:
> 
> https://sites.google.com/site/swiftdevel/sites/pbs/cray
> 
> and the Beagle sub-page.
> 
> Let me know if you get stuck on anything.
> 
> Justin
> 
> On Sun, 16 Oct 2011, David Kelly wrote:
> 
> >
> > Yep - I was in the process of migrating the automated SVN jobs to
> > the swift user, but it looks like it wasn't running correctly due to
> > filesystem permissions. I am manually running the update now. It
> > should be updated within 15 minutes or so.
> >
> > David
> >
> > ----- Original Message -----
> >> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
> >> To: "Michael Wilde" <wilde at mcs.anl.gov>
> >> Cc: "David Kelly" <davidk at ci.uchicago.edu>, "Swift Devel"
> >> <swift-devel at ci.uchicago.edu>
> >> Sent: Sunday, October 16, 2011 12:42:23 PM
> >> Subject: Re: Where is latest doc on running Swift on Beagle? Covers
> >> OpenMP apps?
> >> On Sun, Oct 16, 2011 at 11:15 AM, Michael Wilde < wilde at mcs.anl.gov
> >> >
> >> wrote:
> >>
> >>
> >> David, Ketan,
> >>
> >> I need to run some things on Beagle, asap.
> >>
> >> Ketan, where is the latest and best documentation for this? I see
> >> your
> >> edits below to the 0.93 Site Guide. But I dont see that online
> >> where I
> >> would expect it:
> >>
> >> http://www.ci.uchicago.edu/swift/wwwdev/guides/release-0.93/siteguide/siteguide.html#_beagle
> >>
> >> David, is it just that this document is not being correctly pushed
> >> to
> >> the wwwdev site on a nightly basis?
> >>
> >>
> >>
> >> That seems to be the case. I have committed a little change just
> >> now,
> >> may be that will trigger a doc build. The link you mentioned is not
> >> the latest for Swift on Beagle. See this one which has
> >> documentation
> >> for scaling up runs on Beagle:
> >>
> >>
> >> http://www.ci.uchicago.edu/~ketan/swift-docs/release-0.93/siteguide/siteguide.html#_beagle
> >>
> >>
> >>
> >> Ketan, is the latest info on running Swift on Beagle now all in the
> >> siteguide? Is the info you were putting in the cookbook (I see many
> >> commits there) now all consolidated into the Site Guide? And is
> >> there
> >> a difference in sites.xml settings between 0.93 and trunk? Lastly,
> >> which release works best?
> >>
> >>
> >>
> >> Yes, the sitesguide for release-0.93 is the latest on Swift Beagle
> >> documentation. My cookbook info is all consolidated on sitesguide.
> >> There is no difference between sites file for 0.93 and trunk.
> >>
> >>
> >> Regards,
> >> Ketan
> >>
> >>
> >>
> >>
> >> Second question: I need to run a script that executes many 24-core
> >> OpenMP apps. Is the necessary support for this in 0.93? What if any
> >> declarations do I need other than to say jobsPerNode=1? Glen, are
> >> you
> >> running OpenMP on Beagle and if so what release and sites file are
> >> you
> >> using?
> >>
> >> Im assuming Justin's latest changes to sites.xml are in trunk but
> >> not
> >> 0.93? If that is correct, is there a corresponding site site for
> >> Beagle for trunk?
> >>
> >> Thanks,
> >>
> >> - Mike
> >>
> >>
> >> ----- Forwarded Message -----
> >> From: ketan at ci.uchicago.edu
> >> To: swift-commit at ci.uchicago.edu
> >> Sent: Sunday, September 18, 2011 10:14:10 PM
> >> Subject: [Swift-commit] r5126 -
> >> branches/release-0.93/docs/siteguide
> >>
> >> Author: ketan
> >> Date: 2011-09-18 22:14:10 -0500 (Sun, 18 Sep 2011)
> >> New Revision: 5126
> >>
> >> Modified:
> >> branches/release-0.93/docs/siteguide/beagle
> >> Log:
> >> added content to beagle siteguide
> >>
> >> Modified: branches/release-0.93/docs/siteguide/beagle
> >> ===================================================================
> >> --- branches/release-0.93/docs/siteguide/beagle 2011-09-19 02:41:02
> >> UTC (rev 5125)
> >> +++ branches/release-0.93/docs/siteguide/beagle 2011-09-19 03:14:10
> >> UTC (rev 5126)
> >> @@ -52,9 +52,38 @@
> >> A key factor in scaling up Swift runs on Beagle is to setup the
> >> sites.xml parameters.
> >> The following sites.xml parameters must be set to scale that is
> >> intended for a large run:
> >>
> >> - * walltime: The expected walltime for completion of your run.
> >> This
> >> parameter is accepted in seconds.
> >> - * slots: Number of qsub jobs needs to be submitted by swift. This
> >> number will determine how many qsubs swift will submit for your
> >> run.
> >> Typical values range between 40 and 80 for large runs.
> >> - * nodegranularity: Determines the number of nodes per job. Total
> >> nodes will thus be slots times nodegranularity. This may vary for
> >> advanced configurations though.
> >> - * maxnodes: Determines the maximum number of nodes a job must
> >> pack
> >> into its qsub. This parameter determines the largest single job
> >> that
> >> your run will submit.
> >> + * *maxTime* : The expected walltime for completion of your run.
> >> This
> >> parameter is accepted in seconds.
> >> + * *slots* : Number of qsub jobs needs to be submitted by swift.
> >> This
> >> number will determine how many qsubs swift will submit for your
> >> run.
> >> Typical values range between 40 and 80 for large runs.
> >> + * *nodeGranularity* : Determines the number of nodes per job.
> >> Total
> >> nodes will thus be slots times nodegranularity. This may vary for
> >> advanced configurations though.
> >> + * *maxNodes* : Determines the maximum number of nodes a job must
> >> pack into its qsub. This parameter determines the largest single
> >> job
> >> that your run will submit.
> >> + * *jobThrottle* : A factor that determines the number of tasks
> >> dispatched simultaneously. The intended number of simultaneous
> >> tasks
> >> must match the number of cores targeted. The number of tasks is
> >> calculated from the jobThrottle factor is as follows:
> >>
> >> +----
> >> +Number of Tasks = (JobThrottle x 100) + 1
> >> +----
> >>
> >> +Following is an example sites.xml for a 50 slots run with each
> >> slot
> >> occupying 4 nodes (thus, a 200 node run):
> >> +
> >> +-----
> >> +<config>
> >> + <pool handle="pbs">
> >> + <execution provider="coaster" jobmanager="local:pbs"/>
> >> + <profile namespace="globus" key="project">CI-CCR000013</profile>
> >> +
> >> + <profile namespace="globus" key="ppn">24:cray:pack</profile>
> >> +
> >> + <profile namespace="globus" key="jobsPerNode">24</profile>
> >> + <profile namespace="globus" key="maxTime">50000</profile>
> >> + <profile namespace="globus" key="slots">50</profile>
> >> + <profile namespace="globus" key="nodeGranularity">4</profile>
> >> + <profile namespace="globus" key="maxNodes">4</profile>
> >> +
> >> + <profile namespace="karajan" key="jobThrottle">48.00</profile>
> >> + <profile namespace="karajan" key="initialScore">10000</profile>
> >> +
> >> + <filesystem provider="local"/>
> >> + <workdirectory
> >> >/lustre/beagle/ketan/swift.workdir</workdirectory>
> >> + </pool>
> >> +</config>
> >> +-----
> >> +
> >>
> >> _______________________________________________
> >> Swift-commit mailing list
> >> Swift-commit at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-commit
> >>
> >> --
> >> Michael Wilde
> >> Computation Institute, University of Chicago
> >> Mathematics and Computer Science Division
> >> Argonne National Laboratory
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Ketan
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
> 
> --
> Justin M Wozniak
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory