[Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?

Tue Oct 18 08:51:15 CDT 2011

On Mon, 17 Oct 2011, Michael Wilde wrote:

> Justin, I have a few questions about this page:
>
> "If using Coasters, the count attribute is the number of running 
> worker.pl scripts (1); this is the number of nodes to use."
>
> What is the "count" attribute?

That's an internally-used attribute- I will clarify this.

> "The ppn attribute is available but should be left to default to 1."
>
> This is because we are running one program per node, worker.pl, and 
> giving it (typically) the full "depth" of the node (eg 24 cores) to 
> manage?

Yes.

>  <profile namespace="globus" key="providerAttributes">
>    pbs.aprun;pbs.mpp;depth=24
>  </profile>
>
> I find this attribute confusing. I assume this means:
>
> pbs.aprun: generate an aprun command for Crays in the pbs submit file.

Yes.

The provider attributes concept is described here:

https://sites.google.com/site/swiftdevel/internals/providers/coasters-provider

> pbs.mpp: generate Cray-style mppwidth and mppnppn attributes?

> Of what value? Can a value other than 1 be specified?

This takes the normal width/ppn attributes and uses them in 
mppwidth/mppnppn format.

> depth= : why does 
> depth not have a pbs.depth prefix?

I thought we would leave the depth unprefixed because other schedulers 
could conceivably have depth.

> ppn: how would that be specified?

Use attribute "ppn".  I can add a note about this.

> Leave off ppn only for Cray? What other PBS attributes can be specified, 
> and how are they processed?

Leave it to default to 1.  If you need additional settings, let me know.

> On the PBS page above the Cray page you say:
>
> <profile namespace="globus" key=" ... "> ... </profile>
> pbs.properties: Adds the value to the end of the "#PBS -l " line.  See Fusion for an example use case
> pbs.mpp: If value is set, use mppwidth/mppnppn instead of nodes/ppn in PBS submit file
> pbs.aprun: If value is set, use aprun-based command line
> pbs.resources: Adds the value to a new "#PBS -l" line.
>
> Is the key always providerAttributes?

See the provider attributes page.

> pbs.mpp by "if value is set" you mean "if this string is present in the 
> tags value, separated from other strings by semicolons"?

Idem.

> pbs.properties and pbs.resources - are the same except for where they 
> are inserted?  Are these followed by a pbs -l attributes such as 
> "nodes=10:ppn=4"?

Yes.

> I think several examples are needed to be able to understand how to use 
> (and document) these.
>
> Could you elaborate a bit in the swiftdevel page?  Or are these 
> documented already elsewhere?

Sure.

>
> ----- Original Message -----
>> From: "Justin M Wozniak" <wozniak at mcs.anl.gov>
>> To: "David Kelly" <davidk at ci.uchicago.edu>
>> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>
>> Sent: Monday, October 17, 2011 10:02:15 AM
>> Subject: Re: [Swift-devel] Where is latest doc on running Swift on Beagle? Covers OpenMP apps?
>> My notes about Beagle are at:
>>
>> https://sites.google.com/site/swiftdevel/sites/pbs/cray
>>
>> and the Beagle sub-page.
>>
>> Let me know if you get stuck on anything.
>>
>> Justin
>>
>> On Sun, 16 Oct 2011, David Kelly wrote:
>>
>>>
>>> Yep - I was in the process of migrating the automated SVN jobs to
>>> the swift user, but it looks like it wasn't running correctly due to
>>> filesystem permissions. I am manually running the update now. It
>>> should be updated within 15 minutes or so.
>>>
>>> David
>>>
>>> ----- Original Message -----
>>>> From: "Ketan Maheshwari" <ketancmaheshwari at gmail.com>
>>>> To: "Michael Wilde" <wilde at mcs.anl.gov>
>>>> Cc: "David Kelly" <davidk at ci.uchicago.edu>, "Swift Devel"
>>>> <swift-devel at ci.uchicago.edu>
>>>> Sent: Sunday, October 16, 2011 12:42:23 PM
>>>> Subject: Re: Where is latest doc on running Swift on Beagle? Covers
>>>> OpenMP apps?
>>>> On Sun, Oct 16, 2011 at 11:15 AM, Michael Wilde < wilde at mcs.anl.gov
>>>>>
>>>> wrote:
>>>>
>>>>
>>>> David, Ketan,
>>>>
>>>> I need to run some things on Beagle, asap.
>>>>
>>>> Ketan, where is the latest and best documentation for this? I see
>>>> your
>>>> edits below to the 0.93 Site Guide. But I dont see that online
>>>> where I
>>>> would expect it:
>>>>
>>>> http://www.ci.uchicago.edu/swift/wwwdev/guides/release-0.93/siteguide/siteguide.html#_beagle
>>>>
>>>> David, is it just that this document is not being correctly pushed
>>>> to
>>>> the wwwdev site on a nightly basis?
>>>>
>>>>
>>>>
>>>> That seems to be the case. I have committed a little change just
>>>> now,
>>>> may be that will trigger a doc build. The link you mentioned is not
>>>> the latest for Swift on Beagle. See this one which has
>>>> documentation
>>>> for scaling up runs on Beagle:
>>>>
>>>>
>>>> http://www.ci.uchicago.edu/~ketan/swift-docs/release-0.93/siteguide/siteguide.html#_beagle
>>>>
>>>>
>>>>
>>>> Ketan, is the latest info on running Swift on Beagle now all in the
>>>> siteguide? Is the info you were putting in the cookbook (I see many
>>>> commits there) now all consolidated into the Site Guide? And is
>>>> there
>>>> a difference in sites.xml settings between 0.93 and trunk? Lastly,
>>>> which release works best?
>>>>
>>>>
>>>>
>>>> Yes, the sitesguide for release-0.93 is the latest on Swift Beagle
>>>> documentation. My cookbook info is all consolidated on sitesguide.
>>>> There is no difference between sites file for 0.93 and trunk.
>>>>
>>>>
>>>> Regards,
>>>> Ketan
>>>>
>>>>
>>>>
>>>>
>>>> Second question: I need to run a script that executes many 24-core
>>>> OpenMP apps. Is the necessary support for this in 0.93? What if any
>>>> declarations do I need other than to say jobsPerNode=1? Glen, are
>>>> you
>>>> running OpenMP on Beagle and if so what release and sites file are
>>>> you
>>>> using?
>>>>
>>>> Im assuming Justin's latest changes to sites.xml are in trunk but
>>>> not
>>>> 0.93? If that is correct, is there a corresponding site site for
>>>> Beagle for trunk?
>>>>
>>>> Thanks,
>>>>
>>>> - Mike
>>>>
>>>>
>>>> ----- Forwarded Message -----
>>>> From: ketan at ci.uchicago.edu
>>>> To: swift-commit at ci.uchicago.edu
>>>> Sent: Sunday, September 18, 2011 10:14:10 PM
>>>> Subject: [Swift-commit] r5126 -
>>>> branches/release-0.93/docs/siteguide
>>>>
>>>> Author: ketan
>>>> Date: 2011-09-18 22:14:10 -0500 (Sun, 18 Sep 2011)
>>>> New Revision: 5126
>>>>
>>>> Modified:
>>>> branches/release-0.93/docs/siteguide/beagle
>>>> Log:
>>>> added content to beagle siteguide
>>>>
>>>> Modified: branches/release-0.93/docs/siteguide/beagle
>>>> ===================================================================
>>>> --- branches/release-0.93/docs/siteguide/beagle 2011-09-19 02:41:02
>>>> UTC (rev 5125)
>>>> +++ branches/release-0.93/docs/siteguide/beagle 2011-09-19 03:14:10
>>>> UTC (rev 5126)
>>>> @@ -52,9 +52,38 @@
>>>> A key factor in scaling up Swift runs on Beagle is to setup the
>>>> sites.xml parameters.
>>>> The following sites.xml parameters must be set to scale that is
>>>> intended for a large run:
>>>>
>>>> - * walltime: The expected walltime for completion of your run.
>>>> This
>>>> parameter is accepted in seconds.
>>>> - * slots: Number of qsub jobs needs to be submitted by swift. This
>>>> number will determine how many qsubs swift will submit for your
>>>> run.
>>>> Typical values range between 40 and 80 for large runs.
>>>> - * nodegranularity: Determines the number of nodes per job. Total
>>>> nodes will thus be slots times nodegranularity. This may vary for
>>>> advanced configurations though.
>>>> - * maxnodes: Determines the maximum number of nodes a job must
>>>> pack
>>>> into its qsub. This parameter determines the largest single job
>>>> that
>>>> your run will submit.
>>>> + * *maxTime* : The expected walltime for completion of your run.
>>>> This
>>>> parameter is accepted in seconds.
>>>> + * *slots* : Number of qsub jobs needs to be submitted by swift.
>>>> This
>>>> number will determine how many qsubs swift will submit for your
>>>> run.
>>>> Typical values range between 40 and 80 for large runs.
>>>> + * *nodeGranularity* : Determines the number of nodes per job.
>>>> Total
>>>> nodes will thus be slots times nodegranularity. This may vary for
>>>> advanced configurations though.
>>>> + * *maxNodes* : Determines the maximum number of nodes a job must
>>>> pack into its qsub. This parameter determines the largest single
>>>> job
>>>> that your run will submit.
>>>> + * *jobThrottle* : A factor that determines the number of tasks
>>>> dispatched simultaneously. The intended number of simultaneous
>>>> tasks
>>>> must match the number of cores targeted. The number of tasks is
>>>> calculated from the jobThrottle factor is as follows:
>>>>
>>>> +----
>>>> +Number of Tasks = (JobThrottle x 100) + 1
>>>> +----
>>>>
>>>> +Following is an example sites.xml for a 50 slots run with each
>>>> slot
>>>> occupying 4 nodes (thus, a 200 node run):
>>>> +
>>>> +-----
>>>> +<config>
>>>> + <pool handle="pbs">
>>>> + <execution provider="coaster" jobmanager="local:pbs"/>
>>>> + <profile namespace="globus" key="project">CI-CCR000013</profile>
>>>> +
>>>> + <profile namespace="globus" key="ppn">24:cray:pack</profile>
>>>> +
>>>> + <profile namespace="globus" key="jobsPerNode">24</profile>
>>>> + <profile namespace="globus" key="maxTime">50000</profile>
>>>> + <profile namespace="globus" key="slots">50</profile>
>>>> + <profile namespace="globus" key="nodeGranularity">4</profile>
>>>> + <profile namespace="globus" key="maxNodes">4</profile>
>>>> +
>>>> + <profile namespace="karajan" key="jobThrottle">48.00</profile>
>>>> + <profile namespace="karajan" key="initialScore">10000</profile>
>>>> +
>>>> + <filesystem provider="local"/>
>>>> + <workdirectory
>>>>> /lustre/beagle/ketan/swift.workdir</workdirectory>
>>>> + </pool>
>>>> +</config>
>>>> +-----
>>>> +
>>>>
>>>> _______________________________________________
>>>> Swift-commit mailing list
>>>> Swift-commit at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-commit
>>>>
>>>> --
>>>> Michael Wilde
>>>> Computation Institute, University of Chicago
>>>> Mathematics and Computer Science Division
>>>> Argonne National Laboratory
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ketan
>>> _______________________________________________
>>> Swift-devel mailing list
>>> Swift-devel at ci.uchicago.edu
>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>
>> --
>> Justin M Wozniak
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
>

-- 
Justin M Wozniak