<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'><div>It does seem to be something on the Swift side that is throttling somehow, only one submit file is being written per second.</div><div><br></div><div><div>013-09-09 21:41:42,670-0500 DEBUG AbstractExecutor Wrote SGE script to /clhome/WILDE/.globus/scripts/SGE7757422543183944116.submit</div><div>2013-09-09 21:41:44,002-0500 DEBUG AbstractExecutor Wrote SGE script to /clhome/WILDE/.globus/scripts/SGE7192137066159443413.submit</div><div>2013-09-09 21:41:45,040-0500 DEBUG AbstractExecutor Wrote SGE script to /clhome/WILDE/.globus/scripts/SGE906390723669036085.submit</div><div>2013-09-09 21:41:46,075-0500 DEBUG AbstractExecutor Wrote SGE script to /clhome/WILDE/.globus/scripts/SGE4921900111206072571.submit</div></div><div><br></div><div>It could be something with the SGE provider. I'll take a look and see if I can find anything.</div><div><br></div>I also remember seeing the setting below in the user guide. I've never tried changing this value before, but it might be worth a quick try:<div><br></div><div><div class="paragraph" style="font-family: Georgia, serif; font-size: medium;"><p style="margin-top: 0.5em; margin-bottom: 0.5em;">throttle.host.submit</p></div><div class="literalblock" style="margin-top: 1em; margin-bottom: 1.5em; font-family: Georgia, serif; font-size: medium;"><div class="content" style="padding: 0px;"><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><tt style="font-size: inherit; color: navy;">Valid values: <int>, off</tt></pre></div></div><div class="literalblock" style="margin-top: 1em; margin-bottom: 1.5em; font-family: Georgia, serif; font-size: medium;"><div class="content" style="padding: 0px;"><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><tt style="font-size: inherit; color: navy;">Default value: 2</tt></pre></div></div><div class="literalblock" style="margin-top: 1em; margin-bottom: 1.5em; font-family: Georgia, serif; font-size: medium;"><div class="content" style="padding: 0px;"><pre style="padding: 0px; margin-top: 0px; margin-bottom: 0px;"><tt style="font-size: inherit; color: navy;">Limits the number of concurrent submissions for any of the sites
Swift will try to send jobs to. In other words it guarantees that no
more than the value of this throttle jobs sent to any site will be
concurrently in a state of being submitted.</tt></pre></div></div><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Michael Wilde" <wilde@mcs.anl.gov><br><b>To: </b>"Swift Devel" <swift-devel@ci.uchicago.edu><br><b>Sent: </b>Tuesday, September 10, 2013 10:31:40 AM<br><b>Subject: </b>[Swift-devel] Why does SGE provider only start one job per second?<br><br>Running on the SGE systems orthros, when I configure a sites file for 256 single-core workers, I see the workers starting one per second. It seems that the SGE jobs themselves are only emitted one per second.<br><br>A log of a typical run is in:<br> http://www.ci.uchicago.edu/~wilde/IndexStrain-20130909-2141-zq403y5f.log<br><br>It starts off like this:<br><br>2013-09-09 21:41:40,195-0500 INFO RuntimeStats$ProgressTicker <br>2013-09-09 21:41:43,078-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:99<br>2013-09-09 21:41:44,428-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:98 Active:1<br>2013-09-09 21:41:45,476-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:97 Active:2<br>2013-09-09 21:41:46,500-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:96 Active:3<br>2013-09-09 21:41:47,814-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:95 Active:4<br>2013-09-09 21:41:49,643-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:93 Active:6<br>2013-09-09 21:41:50,686-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:92 Active:7<br>2013-09-09 21:41:51,704-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:91 Active:8<br>2013-09-09 21:41:52,772-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:90 Active:9<br>2013-09-09 21:41:53,807-0500 INFO RuntimeStats$ProgressTicker Stage in:1 Submitted:89 Active:10<br><br>In this case the Swift script consists of 100 independent apps emitted right away from a foreach loop.<br><br>As you can see above and in the log, the 100 apps become active at a 1-per-second rate.<br><br>Can anyone tell me whats throttling the app launches in this fashion?<br><br>I dont recall this happening elsewhere, so perhaps its an artifact of the SGE provider.<br><br>The sites.xml file is below.<br><br>Thanks,<br><br>- Mike<br><br><br><config><br><br> <pool handle="localhost"><br> <execution provider="local"/><br> <filesystem provider="local"/><br> <workdirectory>{env.HOME}/swiftwork</workdirectory><br> </pool><br><br> <pool handle="cluster"><br> <execution provider="coaster" jobmanager="local:sge"/><br><br> <!-- Set partition and account here: --><br> <profile namespace="globus" key="queue">sec1all.q</profile> --><br> <profile namespace="globus" key="pe">sec1_all</profile> --><br> <profile namespace="globus" key="ppn">1</profile><br> <!-- <profile namespace="globus" key="project">pi-wilde</profile> --><br><br> <!-- Set number of jobs and nodes per job here: --><br> <profile namespace="globus" key="slots">320</profile><br> <profile namespace="globus" key="maxnodes">1</profile><br> <profile namespace="globus" key="nodegranularity">1</profile><br> <profile namespace="globus" key="jobsPerNode">1</profile> <!-- apps per node! --><br> <profile namespace="karajan" key="jobThrottle">4.00</profile> <!-- eg .11 -> 12 --><br><br> <!-- Set estimated app time (maxwalltime) and requested job time (maxtime) here: --><br> <profile namespace="globus" key="maxWalltime">00:10:00</profile><br> <profile namespace="globus" key="maxtime">200000</profile> <!-- in seconds! --><br><br> <!-- Set data staging model and work dir here: --><br> <filesystem provider="local"/><br> <workdirectory>/tmp/wilde/swiftwork</workdirectory><br><br> <!-- Typically leave these constant: --><br> <!-- <profile namespace="globus" key="slurm.exclusive">false</profile> --><br> <profile namespace="globus" key="highOverAllocation">100</profile><br> <profile namespace="globus" key="lowOverAllocation">100</profile><br> <profile namespace="karajan" key="initialScore">10000</profile><br> </pool><br><br></config><br>_______________________________________________<br>Swift-devel mailing list<br>Swift-devel@ci.uchicago.edu<br>https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel<br></blockquote><br></div></div></body></html>