<div dir="ltr">I do not see any logs in ~/.globus/coasters; yes, /home is mounted on service nodes and is writable from there.<div><br></div><div>I added "--mode script" as a default arg to qsub in provider code, but still getting the same error. Attached is the new log.</div><div><br></div><div>About the manual option, would we also need coaster service to be running? Or just invoking worker would suffice (for troubleshooting purposes)?</div><div><br></div><div>--Ketan</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 2, 2015 at 6:25 PM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="im HOEnZb">On Mon, 2015-03-02 at 18:11 -0600, Ketan Maheshwari wrote:<br>
> I tried this option but did not seem to work. Attached is the log.<br>
<br>
</span><span class="im HOEnZb">Check /home/ketan/.globus/coasters for worker logs. If there aren't any,<br>
it means that <a href="http://worker.pl" target="_blank">worker.pl</a> isn't being started (I'm assuming that /home is<br>
mounted on compute/service nodes).<br>
<br>
If that's the case, I would suggest troubleshooting by manually running<br>
the qsub command and seeing why the worker doesn't start.<br>
<br>
Mihael<br>
<br>
><br>
</span><div class="HOEnZb"><div class="h5">> On Mon, Mar 2, 2015 at 5:27 PM, Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
><br>
> > It would really be much more useful if you posted the full log.<br>
> ><br>
> > Anyway, I believe that what you need to do is:<br>
> > site.cluster.execution.options.workerLoggingLevel = "DEBUG"<br>
> ><br>
> > Mihael<br>
> ><br>
> > On Mon, 2015-03-02 at 16:37 -0600, Ketan Maheshwari wrote:<br>
> > > The qsub command from the log says:<br>
> > ><br>
> > > qsub -e WORKER_LOGGING_LEVEL=NONE --proccount 32 -n 32 -t 40 --cwd ...<br>
> > ><br>
> > > So, the env variable on swift.conf does not seem to take effect.<br>
> > ><br>
> > > On Mon, Mar 2, 2015 at 4:33 PM, Hategan-Marandiuc, Philip M. <<br>
> > > <a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > > > Well, we need to figure out why. Since the qsub command line is in the<br>
> > > > swift log, and the qsub command line should reflect the setting, it<br>
> > > > would be useful if you posted the swift log.<br>
> > > ><br>
> > > > Mihael<br>
> > > ><br>
> > > > On Mon, 2015-03-02 at 16:27 -0600, Ketan Maheshwari wrote:<br>
> > > > > For workerlogs, I am trying:<br>
> > > > ><br>
> > > > > app.bgsh {<br>
> > > > > executable: "/home/ketan/SwiftApps/subjobs/bg.sh"<br>
> > > > > maxWallTime: "00:04:00"<br>
> > > > > env.ENABLE_WORKER_LOGGING="TRUE"<br>
> > > > > env.WORKER_LOGGING_LEVEL="DEBUG"<br>
> > > > > env.WORKER_LOG_DIR="/home/ketan/workerlogs"<br>
> > > > > }<br>
> > > > ><br>
> > > > > Does not seem to trigger logging.<br>
> > > > ><br>
> > > > > Thanks,<br>
> > > > > Ketan<br>
> > > > ><br>
> > > > > On Mon, Mar 2, 2015 at 4:07 PM, Hategan-Marandiuc, Philip M. <<br>
> > > > > <a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
> > > > ><br>
> > > > > > I would recommend enabling worker logging to see if we get any info<br>
> > > > from<br>
> > > > > > the worker process. Could be some simple thing, like the wrong IP<br>
> > > > > > address.<br>
> > > > > ><br>
> > > > > > Mihael<br>
> > > > > ><br>
> > > > > > On Mon, 2015-03-02 at 15:47 -0600, Ketan Maheshwari wrote:<br>
> > > > > > > I trying to run on BG/Q with local:cobalt with trunk but Swift<br>
> > > > crashes<br>
> > > > > > with<br>
> > > > > > > the following error:<br>
> > > > > > ><br>
> > > > > > > Caused by: Exception in bgsh:<br>
> > > > > > > Arguments:<br>
> > > > [/home/ketan/SwiftApps/subjobs/mpicatsnsleep/mpicatnap,<br>
> > > > > > > /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./data.txt,<br>
> > > > > > ><br>
> > > > > ><br>
> > > ><br>
> > /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./outdir/f.0002.out,<br>
> > > > > > > 1]<br>
> > > > > > > Host: cluster<br>
> > > > > > > Directory: catsnsleepmpi-run001/jobs/b/bgsh-3nq3uc5m<br>
> > > > > > > exception @ swift-int-staging.k, line: 165<br>
> > > > > > > Caused by:<br>
> > > > > > > exception @ swift-int-staging.k, line: 160<br>
> > > > > > > Caused by: Block task failed: 0302-2109420-000000 Block task<br>
> > ended<br>
> > > > > > > prematurely<br>
> > > > > > ><br>
> > > > > > > In the log, I see the qsub call being made and a jobid is<br>
> > returned.<br>
> > > > > > > However, I could not figure what is the cause for the task to<br>
> > fail.<br>
> > > > > > ><br>
> > > > > > > One more thing I noticed when translating from old sites conf to<br>
> > new<br>
> > > > is<br>
> > > > > > > that the new conf did not accept the property "globus:mode =<br>
> > script".<br>
> > > > > > ><br>
> > > > > > > A full run log is attached. Thanks for any suggestions.<br>
> > > > > > ><br>
> > > > > > > Thanks,<br>
> > > > > > > Ketan<br>
> > > > > > > _______________________________________________<br>
> > > > > > > Swift-user mailing list<br>
> > > > > > > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > > > > > ><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Swift-user mailing list<br>
> > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> ><br>
<br>
<br>
_______________________________________________<br>
Swift-user mailing list<br>
<a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
<a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
</div></div></blockquote></div><br></div>