<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Justin, I see that Yadu provided tested configurations for running
with Swift 0.94.1, 0.95RC7, and trunk, in his email to this list on
9/25 (pasted below).<br>
<br>
He pointed you to this directory for a sites.xml example for 0.94
and 0.95:<br>
<a class="moz-txt-link-freetext" href="http://users.rcc.uchicago.edu/~yadunand/blacklight-sanity/0.94configs/sites.xml">http://users.rcc.uchicago.edu/~yadunand/blacklight-sanity/0.94configs/sites.xml</a><br>
<br>
The config he provided for 0.94.1 should also work for 0.95RC6
(which I see you are using) <br>
<br>
(Note that we will be posting an 0.95 final release, or an RC7, by
the end of this week).<br>
<br>
Based on this discussion so far, I think the following is a good
base for a sites entry for running on Blacklight:<br>
<br>
<meta charset="utf-8">
<div class="line" style="color: rgb(0, 0, 0); font-family:
monospace; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="webkit-html-tag"><config<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">xmlns</span>="<span
class="webkit-html-attribute-value"><a class="moz-txt-link-freetext" href="http://www.ci.uchicago.edu/swift/SwiftSites">http://www.ci.uchicago.edu/swift/SwiftSites</a></span>"</span>></span></div>
<div class="collapsible-content" style="margin-left: 1em; color:
rgb(0, 0, 0); font-family: monospace; font-size: 13px; font-style:
normal; font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: auto; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="text"></span>
<div class="collapsible" id="collapsible1">
<div class="expanded">
<div class="line"><span class="button collapse-button"
style="-webkit-user-select: none; cursor: pointer;
display: inline-block; margin-left: -10px; width: 10px;
vertical-align: bottom; height: 10px; background-image:
-webkit-canvas(arrowDown); background-position: 0% 0%;
background-repeat: no-repeat;"></span><span
class="webkit-html-tag"><pool<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">handle</span>="<span
class="webkit-html-attribute-value">blacklight</span>"</span>></span></div>
<div class="collapsible-content" style="margin-left: 1em;"><span
class="text"></span>
<div class="line"><span class="webkit-html-tag"><execution<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">provider</span>="<span
class="webkit-html-attribute-value">coaster</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">jobmanager</span>="<span
class="webkit-html-attribute-value">local:pbs</span>"</span>/></span></div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">globus</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">queue</span>"</span>></span><span
class="text">debug</span><span class="webkit-html-tag"></profile><br>
</span><span class="webkit-html-tag"><profile
namespace="karajan"
key="jobThrottle">.320</profile></span><br>
</div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">karajan</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">initialScore</span>"</span>></span><span
class="text">10000</span><span class="webkit-html-tag"></profile></span></div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">globus</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">jobsPerNode</span>"</span>></span><span
class="text">16</span><span class="webkit-html-tag"></profile></span></div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">globus</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">maxtime</span>"</span>></span><span
class="text">900</span><span class="webkit-html-tag"></profile></span></div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">globus</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">maxwalltime</span>"</span>></span><span
class="text">00:10:00</span><span
class="webkit-html-tag"></profile></span></div>
<span class="text"></span><span class="webkit-html-tag"><profile<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">namespace</span>="<span
class="webkit-html-attribute-value">globus</span>"</span><span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">key</span>="<span
class="webkit-html-attribute-value">ppn</span>"</span>></span><span
class="text">16</span><span class="webkit-html-tag"></profile></span><span
class="text"></span>
<div class="line"><span class="webkit-html-tag"><workdirectory></span><span
class="text">/usr/users/8/yadunand/swiftwork</span><span
class="webkit-html-tag"></workdirectory></span></div>
<span class="text"></span>
<div class="line"><span class="webkit-html-tag"><filesystem<span
class="webkit-html-attribute"><span
class="Apple-converted-space"> </span><span
class="webkit-html-attribute-name">provider</span>="<span
class="webkit-html-attribute-value">local</span>"</span>/></span></div>
<span class="text"></span></div>
<div class="line"><span class="webkit-html-tag"></pool></span></div>
</div>
</div>
<span class="text"></span></div>
<div class="line" style="color: rgb(0, 0, 0); font-family:
monospace; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span
class="webkit-html-tag"></config><br>
<br>
for the "queue" tag, use either debug or batch.<br>
<br>
for the "workdirectory" tag, specify a fully qualified directory
pathname that you can write/create in. Note that you might have
a different $HOME "users/N" dir than Yadu's.<br>
<br>
</span><span class="webkit-html-tag">Please double check the
latest code at the link Yadu provided, to make sure I did not
get anything wrong, above.<br>
<br>
Regards,<br>
<br>
- Mike<br>
<br>
</span><br>
-------- Original Message --------
<table class="moz-email-headers-table" cellpadding="0"
cellspacing="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Subject:
</th>
<td>Re: [Swift-user] question about xsede</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Date: </th>
<td>Thu, 25 Sep 2014 17:25:01 -0500</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">From: </th>
<td>Yadu Nand <a class="moz-txt-link-rfc2396E" href="mailto:yadudoc1729@gmail.com"><yadudoc1729@gmail.com></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">To: </th>
<td>Michael Wilde <a class="moz-txt-link-rfc2396E" href="mailto:wilde@anl.gov"><wilde@anl.gov></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">CC: </th>
<td>Swift User <a class="moz-txt-link-rfc2396E" href="mailto:swift-user@ci.uchicago.edu"><swift-user@ci.uchicago.edu></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Hi Justin,</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Here are some
tested configs and a small README from running the sanity test
on Blacklight:</div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif"><a moz-do-not-send="true"
href="http://users.rcc.uchicago.edu/%7Eyadunand/blacklight-sanity/">http://users.rcc.uchicago.edu/~yadunand/blacklight-sanity/</a></font><br>
</div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif"><br>
</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif">There's an example each of configs
for Swift 0.94, Swift 0.95 and the configs we would use
going</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif">forward (Swift 0.96 and current
trunk) </font><span
style="font-family:arial,helvetica,sans-serif">in that
folder.</span></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif"><br>
</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif">In the example, I've used ppn=16 (or
any multiple of 16) which seems to work as a substitute for
ncpus.</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif"><br>
</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif">Hope that helps!</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif"><br>
</font></div>
<div class="gmail_default" style=""><font face="arial,
helvetica, sans-serif">-Yadu</font></div>
</div>
<br>
</div>
<br>
<div class="moz-cite-prefix">On 9/29/14, 9:24 AM, Michael Wilde
wrote:<br>
</div>
<blockquote cite="mid:54296B97.6060506@anl.gov" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Justin, in your sites entry below, this line looks suspect:<br>
<br>
<workdirectory>/brashear/usrname</workdirectory><br>
<br>
That needs to be the name of a writable directory.<br>
<br>
- Mike<br>
<br>
<br>
<div class="moz-cite-prefix">On 9/28/14, 6:03 PM, Justin bbt
wrote:<br>
</div>
<blockquote
cite="mid:CABubRoAygNQTA2D92szo-UkD=ruhwSfafiAuDm4v4eXRK9Y5dA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div dir="ltr">changing the queue to "batch", I get this
<div><span class="im"
style="font-family:arial,sans-serif;font-size:12.7272720336914px">
<div>Execution failed:</div>
<div>Exception in simulate:</div>
<div> Arguments: [--timesteps, 1, --range, 100,
--nvalues, 5]</div>
<div> Host: black</div>
</span>
<div
style="font-family:arial,sans-serif;font-size:12.7272720336914px">
Directory: p4-run026/jobs/5/simulate-58d6x2yl</div>
<span class="im"
style="font-family:arial,sans-serif;font-size:12.7272720336914px">
<div><span style="white-space:pre-wrap"> </span>exception
@ swift-int-staging.k, line: 181</div>
<div>Caused by: </div>
<div><span style="white-space:pre-wrap"> </span>exception
@ swift-int-staging.k, line: 177</div>
<div>Caused by: null</div>
</span>
<div
style="font-family:arial,sans-serif;font-size:12.7272720336914px">Caused
by:
org.globus.cog.abstraction.impl.common.execution.JobException:
Job failed with an exit code of 1</div>
</div>
<div
style="font-family:arial,sans-serif;font-size:12.7272720336914px"><br>
</div>
<div
style="font-family:arial,sans-serif;font-size:12.7272720336914px">???? </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Sep 27, 2014 at 5:17 PM,
Mihael Hategan <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:hategan@mcs.anl.gov"
target="_blank">hategan@mcs.anl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
Justin,<br>
<br>
Is there a queue named "normal" on that machine (qstat -q
should tell<br>
you)?<br>
<span class="HOEnZb"><font color="#888888"><br>
Mihael<br>
</font></span>
<div class="HOEnZb">
<div class="h5"><br>
On Sat, 2014-09-27 at 15:41 -0400, Justin bbt wrote:<br>
> So, I am using this config<br>
><br>
><br>
> <pool handle="black"><br>
> <execution provider="coaster"
jobmanager="local:pbs" URL="none"/><br>
> <profile namespace="env"
key="PATHPREFIX">{env.PWD}/../app</profile><br>
> <profile namespace="globus"
key="jobsPerNode">1</profile><br>
> <profile namespace="globus"
key="queue">normal</profile><br>
> <profile namespace="globus"
key="ppn">16</profile><br>
> <profile namespace="globus"
key="maxWallTime">00:01:00</profile><br>
> <profile namespace="globus"
key="maxTime">3600</profile><br>
> <profile namespace="globus"
key="lowOverAllocation">100</profile><br>
> <profile namespace="globus"
key="highOverAllocation">100</profile><br>
> <profile namespace="globus"
key="slots">2</profile><br>
> <profile namespace="globus"
key="maxNodes">1</profile><br>
> <profile namespace="globus"
key="nodeGranularity">1</profile><br>
> <profile namespace="karajan"
key="jobThrottle">.320</profile><br>
> <profile namespace="karajan"
key="initialScore">10000</profile><br>
> <profile namespace="globus"
key="slots">1</profile><br>
> <profile namespace="globus"
key="project">TG-CCR134513</profile><br>
>
<workdirectory>/brashear/usrname</workdirectory><br>
> <profile namespace="swift"
key="stagingMethod">local</profile><br>
> <filesystem provider="local" /><br>
> </pool><br>
><br>
> but, it does not work and give this error :<br>
><br>
> Swift 0.95 RC6 swift-r7900 cog-r3908<br>
> RunID: run011<br>
> Warning: The @ syntax for function invocation is
deprecated<br>
> [Error] sites.xml, line 2, col 10: cvc-elt.1:
Cannot find the declaration<br>
> of element 'config'.<br>
> Progress: Sat, 27 Sep 2014 15:35:52-0400<br>
><br>
> Could not submit job (qsub reported an exit code
of 170).<br>
> qsub: Unknown queue MSG=cannot locate queue<br>
><br>
> Execution failed:<br>
> Exception in simulate:<br>
> Arguments: [--timesteps, 1, --range, 100,
--nvalues, 5]<br>
> Host: black<br>
> Directory: p4-run011/jobs/s/simulate-sdm041yl<br>
> exception @ swift-int-staging.k, line: 181<br>
> Caused by:<br>
> exception @ swift-int-staging.k, line: 177<br>
> Caused by: Block task failed: Error submitting
block task<br>
>
org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
Could<br>
> not submit job (qsub reported an exit code of
170).<br>
> qsub: Unknown queue MSG=cannot locate queue<br>
><br>
> at<br>
>
org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63)<br>
> at<br>
>
org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45)<br>
> at<br>
>
org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61)<br>
> at<br>
>
org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:70)<br>
> Caused by:<br>
>
org.globus.cog.abstraction.impl.scheduler.common.ProcessException:
Could<br>
> not submit job (qsub reported an exit code of
170).<br>
> qsub: Unknown queue MSG=cannot locate queue<br>
><br>
> at<br>
>
org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:113)<br>
> at<br>
>
org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:53)<br>
> ... 3 more<br>
><br>
><br>
><br>
> also, in some configs I saw this. But I dont know
what this is and what<br>
> values should I set to<br>
><br>
> <profile namespace="globus"<br>
>
key="providerAttributes">pbs.aprun;pbs.mpp;depth=32</profile><br>
><br>
><br>
> On Tue, Sep 23, 2014 at 6:22 PM, Justin bbt <<a
moz-do-not-send="true"
href="mailto:justinbbt@gmail.com">justinbbt@gmail.com</a>>
wrote:<br>
><br>
> > Thank you very much.<br>
> ><br>
> > I am actually using the Blacklight, which I
guess is PBS based.<br>
> > So, should I use the Crays tutorial and
setting ?<br>
> > <a moz-do-not-send="true"
href="http://swift-lang.org/tutorials/cray/tutorial.html"
target="_blank">http://swift-lang.org/tutorials/cray/tutorial.html</a><br>
> ><br>
> > On Tue, Sep 23, 2014 at 10:31 AM, Ketan
Maheshwari <<a moz-do-not-send="true"
href="mailto:ketan@mcs.anl.gov">ketan@mcs.anl.gov</a>><br>
> > wrote:<br>
> ><br>
> >> Hi Justin,<br>
> >><br>
> >> If you are using xsede Stampede regular
nodes (non xeon phi), here is a<br>
> >> site configuration that has worked for
me in the past, connecting over ssh<br>
> >> to slurm:<br>
> >><br>
> >> <?xml version="1.0"
encoding="UTF-8"?><br>
> >> <config xmlns="<a
moz-do-not-send="true"
href="http://www.ci.uchicago.edu/swift/SwiftSites"
target="_blank">http://www.ci.uchicago.edu/swift/SwiftSites</a>"><br>
> >> <pool handle="stampede"><br>
> >> <execution provider="coaster"
jobmanager="ssh-cl:slurm" url="<br>
> >> <a moz-do-not-send="true"
href="http://stampede.tacc.utexas.edu"
target="_blank">stampede.tacc.utexas.edu</a>"/><br>
> >> <filesystem provider="local" /><br>
> >> <profile namespace="globus"
key="jobsPerNode">1</profile><br>
> >> <profile namespace="globus"
key="ppn">1</profile><br>
> >> <profile namespace="globus"
key="maxTime">7500</profile><br>
> >> <profile namespace="globus"
key="maxwalltime">00:10:00</profile><br>
> >> <profile namespace="globus"
key="lowOverallocation">100</profile><br>
> >> <profile namespace="globus"
key="highOverallocation">100</profile><br>
> >> <profile namespace="globus"
key="queue">normal</profile><br>
> >> <profile namespace="globus"
key="nodeGranularity">1</profile><br>
> >> <profile namespace="globus"
key="maxNodes">1</profile><br>
> >> <profile namespace="globus"
key="slots">1</profile><br>
> >> <profile namespace="globus"
key="project">TG-EAR130015</profile><br>
> >> <profile namespace="karajan"
key="jobThrottle">.3199</profile><br>
> >> <profile namespace="karajan"
key="initialScore">10000</profile><br>
> >>
<workdirectory>/tmp/{env.USER}/swift.work</workdirectory><br>
> >> </pool><br>
> >> </config><br>
> >><br>
> >> You will need to replace project id with
yours.<br>
> >><br>
> >> Thanks,<br>
> >> Ketan<br>
> >><br>
> >> On Mon, Sep 22, 2014 at 8:32 PM, Justin
bbt <<a moz-do-not-send="true"
href="mailto:justinbbt@gmail.com">justinbbt@gmail.com</a>>
wrote:<br>
> >><br>
> >>> Hi<br>
> >>><br>
> >>> If I want to use resources on the
xsede<br>
> >>> <a moz-do-not-send="true"
href="https://www.xsede.org/overview"
target="_blank">https://www.xsede.org/overview</a><br>
> >>> which site config should I use ?<br>
> >>><br>
> >>><br>
> >>>
_______________________________________________<br>
> >>> Swift-user mailing list<br>
> >>> <a moz-do-not-send="true"
href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> >>> <a moz-do-not-send="true"
href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user"
target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> >>><br>
> >><br>
> >><br>
> ><br>
> _______________________________________________<br>
> Swift-user mailing list<br>
> <a moz-do-not-send="true"
href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> <a moz-do-not-send="true"
href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user"
target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
<br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Swift-user mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a></pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Michael Wilde
Mathematics and Computer Science Computation Institute
Argonne National Laboratory The University of Chicago
</pre>
</body>
</html>