<div dir="ltr">OK, I found that <a href="http://worker.pl">worker.pl</a> was crashing because of my subjob related mods. I forgot to declare a variable using "my". After this change, it runs. <div><br></div><div>However, jobs that complete are not reported to be completed; they stay in "active" state as seen from the progress log till the job times out. I also see the following lines in stderr:</div><div><br></div><div><div>Use of uninitialized value $SOFT_IMAGE_JOB_ID in numeric eq (==) at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 2235.</div><div>Use of uninitialized value in concatenation (.) or string at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 387.</div><div>Use of uninitialized value $SOFT_IMAGE_JOB_ID in numeric eq (==) at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 2235.</div><div>Use of uninitialized value $SOFT_IMAGE_JOB_ID in numeric eq (==) at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 2235.</div><div>Use of uninitialized value in concatenation (.) or string at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 387.</div><div>Use of uninitialized value $SOFT_IMAGE_JOB_ID in numeric eq (==) at /home/ketan/.globus/coasters/<a href="http://cscript225276003254762418.pl">cscript225276003254762418.pl</a> line 2235.</div></div><div><br></div><div>Not sure if these are errors or warnings and relevant.</div><div><br></div><div>Attached is the complete log.</div><div><br></div><div>Thanks,</div><div>Ketan</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 2, 2015 at 7:35 PM, Hategan-Marandiuc, Philip M. <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, 2015-03-02 at 18:55 -0600, Ketan Maheshwari wrote:<br>
> I do not see any logs in ~/.globus/coasters; yes, /home is mounted on<br>
> service nodes and is writable from there.<br>
><br>
> I added "--mode script" as a default arg to qsub in provider code, but<br>
> still getting the same error. Attached is the new log.<br>
><br>
> About the manual option, would we also need coaster service to be running?<br>
> Or just invoking worker would suffice (for troubleshooting purposes)?<br>
<br>
</span>Just invoking <a href="http://worker.pl" target="_blank">worker.pl</a>. You should eventually get a log file from the<br>
worker that indicates that the perl process has started. It will fail,<br>
unable to connect to the service, but that's secondary.<br>
<br>
I'm surprised that you are not getting any stdout/stderr from the<br>
process. Maybe the secret is somewhere around that.<br>
<br>
Mihael<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> --Ketan<br>
><br>
> On Mon, Mar 2, 2015 at 6:25 PM, Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
><br>
> > On Mon, 2015-03-02 at 18:11 -0600, Ketan Maheshwari wrote:<br>
> > > I tried this option but did not seem to work. Attached is the log.<br>
> ><br>
> > Check /home/ketan/.globus/coasters for worker logs. If there aren't any,<br>
> > it means that <a href="http://worker.pl" target="_blank">worker.pl</a> isn't being started (I'm assuming that /home is<br>
> > mounted on compute/service nodes).<br>
> ><br>
> > If that's the case, I would suggest troubleshooting by manually running<br>
> > the qsub command and seeing why the worker doesn't start.<br>
> ><br>
> > Mihael<br>
> ><br>
> > ><br>
> > > On Mon, Mar 2, 2015 at 5:27 PM, Mihael Hategan <<a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>><br>
> > wrote:<br>
> > ><br>
> > > > It would really be much more useful if you posted the full log.<br>
> > > ><br>
> > > > Anyway, I believe that what you need to do is:<br>
> > > > site.cluster.execution.options.workerLoggingLevel = "DEBUG"<br>
> > > ><br>
> > > > Mihael<br>
> > > ><br>
> > > > On Mon, 2015-03-02 at 16:37 -0600, Ketan Maheshwari wrote:<br>
> > > > > The qsub command from the log says:<br>
> > > > ><br>
> > > > > qsub -e WORKER_LOGGING_LEVEL=NONE --proccount 32 -n 32 -t 40 --cwd<br>
> > ...<br>
> > > > ><br>
> > > > > So, the env variable on swift.conf does not seem to take effect.<br>
> > > > ><br>
> > > > > On Mon, Mar 2, 2015 at 4:33 PM, Hategan-Marandiuc, Philip M. <<br>
> > > > > <a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
> > > > ><br>
> > > > > > Well, we need to figure out why. Since the qsub command line is in<br>
> > the<br>
> > > > > > swift log, and the qsub command line should reflect the setting, it<br>
> > > > > > would be useful if you posted the swift log.<br>
> > > > > ><br>
> > > > > > Mihael<br>
> > > > > ><br>
> > > > > > On Mon, 2015-03-02 at 16:27 -0600, Ketan Maheshwari wrote:<br>
> > > > > > > For workerlogs, I am trying:<br>
> > > > > > ><br>
> > > > > > > app.bgsh {<br>
> > > > > > > executable: "/home/ketan/SwiftApps/subjobs/bg.sh"<br>
> > > > > > > maxWallTime: "00:04:00"<br>
> > > > > > > env.ENABLE_WORKER_LOGGING="TRUE"<br>
> > > > > > > env.WORKER_LOGGING_LEVEL="DEBUG"<br>
> > > > > > > env.WORKER_LOG_DIR="/home/ketan/workerlogs"<br>
> > > > > > > }<br>
> > > > > > ><br>
> > > > > > > Does not seem to trigger logging.<br>
> > > > > > ><br>
> > > > > > > Thanks,<br>
> > > > > > > Ketan<br>
> > > > > > ><br>
> > > > > > > On Mon, Mar 2, 2015 at 4:07 PM, Hategan-Marandiuc, Philip M. <<br>
> > > > > > > <a href="mailto:hategan@mcs.anl.gov">hategan@mcs.anl.gov</a>> wrote:<br>
> > > > > > ><br>
> > > > > > > > I would recommend enabling worker logging to see if we get any<br>
> > info<br>
> > > > > > from<br>
> > > > > > > > the worker process. Could be some simple thing, like the wrong<br>
> > IP<br>
> > > > > > > > address.<br>
> > > > > > > ><br>
> > > > > > > > Mihael<br>
> > > > > > > ><br>
> > > > > > > > On Mon, 2015-03-02 at 15:47 -0600, Ketan Maheshwari wrote:<br>
> > > > > > > > > I trying to run on BG/Q with local:cobalt with trunk but<br>
> > Swift<br>
> > > > > > crashes<br>
> > > > > > > > with<br>
> > > > > > > > > the following error:<br>
> > > > > > > > ><br>
> > > > > > > > > Caused by: Exception in bgsh:<br>
> > > > > > > > > Arguments:<br>
> > > > > > [/home/ketan/SwiftApps/subjobs/mpicatsnsleep/mpicatnap,<br>
> > > > > > > > ><br>
> > /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./data.txt,<br>
> > > > > > > > ><br>
> > > > > > > ><br>
> > > > > ><br>
> > > ><br>
> > /gpfs/mira-home/ketan/SwiftApps/subjobs/mpicatsnsleep/./outdir/f.0002.out,<br>
> > > > > > > > > 1]<br>
> > > > > > > > > Host: cluster<br>
> > > > > > > > > Directory: catsnsleepmpi-run001/jobs/b/bgsh-3nq3uc5m<br>
> > > > > > > > > exception @ swift-int-staging.k, line: 165<br>
> > > > > > > > > Caused by:<br>
> > > > > > > > > exception @ swift-int-staging.k, line: 160<br>
> > > > > > > > > Caused by: Block task failed: 0302-2109420-000000 Block task<br>
> > > > ended<br>
> > > > > > > > > prematurely<br>
> > > > > > > > ><br>
> > > > > > > > > In the log, I see the qsub call being made and a jobid is<br>
> > > > returned.<br>
> > > > > > > > > However, I could not figure what is the cause for the task to<br>
> > > > fail.<br>
> > > > > > > > ><br>
> > > > > > > > > One more thing I noticed when translating from old sites<br>
> > conf to<br>
> > > > new<br>
> > > > > > is<br>
> > > > > > > > > that the new conf did not accept the property "globus:mode =<br>
> > > > script".<br>
> > > > > > > > ><br>
> > > > > > > > > A full run log is attached. Thanks for any suggestions.<br>
> > > > > > > > ><br>
> > > > > > > > > Thanks,<br>
> > > > > > > > > Ketan<br>
> > > > > > > > > _______________________________________________<br>
> > > > > > > > > Swift-user mailing list<br>
> > > > > > > > > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > > > > > > > ><br>
> > > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > ><br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Swift-user mailing list<br>
> > > > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> > > ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Swift-user mailing list<br>
> > <a href="mailto:Swift-user@ci.uchicago.edu">Swift-user@ci.uchicago.edu</a><br>
> > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user</a><br>
> ><br>
<br>
<br>
</div></div></blockquote></div><br></div>