[Swift-user] Question about nr of nodes

[Jonathan Monette]

It all depends on ho you want to shape your jobs.
slots: The maximum number of coaster blocks to submit(pbs jobs).
maxNodes: The maximum number of nodes in a coaster block
nodeGranularity: How much to increment node count by(nodeGranularity <= maxNodes)

For instance, the following settings:
<profile namespace "globus" key="slots">1</profile>
<profile namespace "globus" key="maxNodes">10</profile>
<profile namespace "globus" key="nodeGranularity">1</profile>

will submit a single coaster block at a time to the system with a node count of anywhere between 1-10.

By changing nodeGranularity to 10 we will get a single coaster block with 10 nodes.  That is the only size block that will be submitted while in the previous example we could get a varying size of coaster blocks.

Adjusting slots will provide more active coaster blocks.

In Glen's example he will requests "$nodes" number of single node jobs.  He could have said the same thing by setting:
<profile namespace "globus" key="slots">1</profile>
<profile namespace "globus" key="maxNodes">$nodes</profile>
<profile namespace "globus" key="nodeGranularity">$nodes</profile>

In his example, several single node coaster blocks would be submitted for execution.  With the above settings, a single multi-node coaster block would be submitted.  If the machine is overloaded and there is slow response time, then Glen's approach would probably be better as the scheduler may bias some single node jobs to run over multi-node jobs to keep the entire machine busy.  This way progress will be made(even if it is slow progress).

Another setting that should be set is:
<profile namespace="globus" key="lowOverallocation">100</profile>
<profile namespace="globus" key="highOverallocation">100</profile>

This will force the coaster blocks to be exactly the maxTime you asked for.  If those are not set coasters dynamically chooses a wall time which is often lower than the time you specified.

I hope all this makes sense.  We are in the process of improving documentation for Swift in general and this is one of the areas we need to better explain.  Please let me know if this all makes sense or anything else I could help with.


On Apr 2, 2012, at 3:57 PM, Glen Hocky wrote:

> What I used to use was (shell variables should be self explanatory). This should limit your number of nodes to $nodes while putting 1 coaster on each one to run your jobs...
> Things may have changed in the last few months and this might not work correctly any more...
> 
>     <profile key="jobsPerNode" namespace="globus">24</profile>
>     <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> 
>     <profile namespace="env" key="OMP_NUM_THREADS">$PPN</profile>
>     <profile namespace="globus" key="maxwalltime">$TIME</profile>
>     <profile namespace="globus" key="maxTime">$MAXTIME</profile>
>     <profile namespace="globus" key="slots">$nodes</profile>
>     <profile namespace="globus" key="nodeGranularity">1</profile>
>     <profile namespace="globus" key="maxNodes">1</profile>
>     <profile namespace="globus" key="lowOverAllocation">100</profile>
>     <profile namespace="globus" key="highOverAllocation">100</profile>
>     <profile namespace="karajan" key="jobThrottle">200.00</profile>
>     <profile namespace="karajan" key="initialScore">10000</profile>
> 
> 
> 
> On Mon, Apr 2, 2012 at 4:54 PM, Jonathan Monette <jonmon at mcs.anl.gov> wrote:
> So you can very nodeGranulairty.  Right now in your sites file you have it set to 1.  That means that coasters will try jobs anywhere in the range 1-50 nodes. If you set nodeGranularity to 50, then coasters will submit a job with 50 nodes.
> 
> On Apr 2, 2012, at 3:48 PM, Lorenzo Pesce wrote:
> 
> > Nth question from me.
> >
> > Do you know of any implicit of explicit limitation in the number of nodes when one submits a job requesting less than 1 hr on Beagle?
> >
> > I asked for 50 nodes (I think) and I got only two.
> >
> > This is my sites file
> >
> > <config>
> >  <pool handle="pbs">
> >    <execution provider="coaster" jobmanager="local:pbs"/>
> >    <!-- replace with your project -->
> >    <profile namespace="globus" key="project">CI-IBN000103</profile>
> >
> >    <profile namespace="globus" key="providerAttributes">pbs.aprun;pbs.mpp;depth=24</profile>
> >
> >    <profile namespace="globus" key="jobsPerNode">24</profile>
> >    <profile namespace="globus" key="maxTime">3600</profile>
> >
> >    <profile namespace="globus" key="slots">1</profile>
> >    <profile namespace="globus" key="nodeGranularity">1</profile>
> >    <profile namespace="globus" key="maxNodes">50</profile>
> >
> >    <profile namespace="karajan" key="jobThrottle">12</profile>
> >    <profile namespace="karajan" key="initialScore">10000</profile>
> >
> >    <filesystem provider="local"/>
> >    <!-- replace this with your home on lustre -->
> >    <workdirectory>/lustre/beagle/GCNet/swift.workdir</workdirectory>
> >  </pool>
> > </config>
> >
> >
> > _______________________________________________
> > Swift-user mailing list
> > Swift-user at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120402/75c569db/attachment.html>