also failed again on ucanl64<div><br></div><div>[...]</div><div><div>Progress: Submitting:1 Submitted:1 Failed but can retry:1</div><div>Failed to transfer wrapper log from AFNIsnr-20090421-1231-4h6oa8k5/info/e on ANLUCTERAGRID64</div>
<div>Progress: Submitting:1 Failed but can retry:2</div><div>Progress: Submitting:1 Failed but can retry:2</div><div>Progress: Submitting:1 Failed but can retry:2</div><div>Progress: Stage in:1 Submitting:1 Failed but can retry:1</div>
<div>Progress: Submitting:1 Submitted:1 Failed but can retry:1</div><div>Progress: Submitting:1 Failed but can retry:2</div><div>Failed to transfer wrapper log from AFNIsnr-20090421-1231-4h6oa8k5/info/g on ANLUCTERAGRID64</div>
<div>Execution failed:</div><div> Exception in AFNI_3dvolreg:</div><div>Arguments: [-twopass, -twodup, -dfile, mot_RFL2.run5_trim, -base, ts.5_trim+orig[92], -prefix, volreg.RFL2.run5_trim, ts.5_trim+orig.BRIK]</div>
<div>Host: ANLUCTERAGRID64</div><div>Directory: AFNIsnr-20090421-1231-4h6oa8k5/jobs/g/AFNI_3dvolreg-gbrmkp9j</div><div>stderr.txt: </div><div><br></div><div>stdout.txt: </div><div><br></div><div>----</div><div><br></div><div>
Caused by:</div><div> Cannot submit job: ; nested exception is: </div><div> java.net.SocketTimeoutException: Read timed out</div><div>gwynn 7%</div><div><br></div><div><br></div></div><div><br></div><div><br>
</div><div><br><br><div class="gmail_quote">On Tue, Apr 21, 2009 at 12:29 PM, Michael Andric <span dir="ltr"><<a href="mailto:andric@uchicago.edu">andric@uchicago.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
just tried on bigred - also failed<div><br></div><div>[...]</div><div><div class="im"><div>Progress: Failed but can retry:3</div><div>Progress: Stage in:1 Failed but can retry:2</div></div><div>Exception occured in the exception handling code, so it cannot be properly propagated to the user</div>
<div>java.net.SocketException: Broken pipe</div><div> at java.net.SocketOutputStream.socketWrite0(Native Method)</div><div> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)</div><div> at java.net.SocketOutputStream.write(SocketOutputStream.java:115)</div>
<div> at java.io.DataOutputStream.writeByte(DataOutputStream.java:136)</div><div> at org.globus.ftp.dc.EBlockImageDCWriter.endOfData(EBlockImageDCWriter.java:63)</div><div> at org.globus.ftp.dc.GridFTPTransferSourceThread.shutdown(GridFTPTransferSourceThread.java:62)</div>
<div> at org.globus.ftp.dc.TransferSourceThread.run(TransferSourceThread.java:87)</div><div class="im"><div>Progress: Failed but can retry:3</div></div><div>Failed to transfer wrapper log from AFNIsnr-20090421-1220-q9598ll1/info/o on BIGRED</div>
<div class="im">
<div>Progress: Failed:1 Failed but can retry:2</div><div>Execution failed:</div><div> Exception in AFNI_3dvolreg:</div></div><div>Arguments: [-twopass, -twodup, -dfile, mot_RFL2.run4_trim, -base, ts.4_trim+orig[92], -prefix, volreg.RFL2.run4_trim, ts.4_trim+orig.BRIK]</div>
<div>Host: BIGRED</div><div>Directory: AFNIsnr-20090421-1220-q9598ll1/jobs/o/AFNI_3dvolreg-od58kp9j</div><div class="im"><div>stderr.txt: </div><div><br></div><div>stdout.txt: </div><div><br></div><div>----</div><div><br>
</div><div>Caused by:</div>
</div><div> Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 451 ocurred during retrieve()</div><div>org.globus.ftp.exception.ServerException: Refusing to start transfer before previous transfer completes (error code 5)</div>
<div>org.globus.ftp.exception.ServerException: Refusing to start transfer before previous transfer completes (error code 5)</div><div> at org.globus.ftp.dc.TransferThreadManager.startTransfer(TransferThreadManager.java:129)</div>
<div> at org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:431)</div><div> at org.globus.ftp.FTPClient.put(FTPClient.java:1289)</div><div> at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImpl.java:427)</div>
<div> at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(DelegatedFileTransferHandler.java:355)</div><div> at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestination(CachingDelegatedFileTransferHandler.java:47)</div>
<div> at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:492)</div><div> at java.lang.Thread.run(Thread.java:595)</div><div>]</div><div>gwynn 4%</div>
<div><div></div><div class="h5">
<br><div class="gmail_quote">On Tue, Apr 21, 2009 at 12:25 PM, Mihael Hategan <span dir="ltr"><<a href="mailto:hategan@mcs.anl.gov" target="_blank">hategan@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ok. My point was, in general, that if there are such catastrophic<br>
failures, spacing out in time a couple of attempts generally raises the<br>
confidence that the problem is not some spooky transient issue.<br>
<div><div></div><div><br>
On Tue, 2009-04-21 at 17:19 +0000, Ben Clifford wrote:<br>
> I can recreate non-swift brokenness on both of those sites.<br>
><br>
> On Tue, 21 Apr 2009, Mihael Hategan wrote:<br>
><br>
> > Did you also try that one again?<br>
> ><br>
> > On Tue, 2009-04-21 at 12:14 -0500, Michael Andric wrote:<br>
> > > what about ucanl?<br>
> > ><br>
> > > On Tue, Apr 21, 2009 at 12:11 PM, Ben Clifford <<a href="mailto:benc@hawaga.org.uk" target="_blank">benc@hawaga.org.uk</a>><br>
> > > wrote:<br>
> > ><br>
> > > On Tue, 21 Apr 2009, Michael Andric wrote:<br>
> > ><br>
> > ><br>
> > > > /disks/ci-gpfs/fmri/cnari/config/sites_bsd.xml<br>
> > ><br>
> > ><br>
> > > I cannot make a manual submission to <a href="http://gwynn.bsd.uchicago.edu" target="_blank">gwynn.bsd.uchicago.edu</a><br>
> > > using<br>
> > > globus-job-run, so this is not a Swift problem. I think this<br>
> > > is something<br>
> > > that support@ci should deal with.<br>
> > ><br>
> > > --<br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > Swift-devel mailing list<br>
> > > <a href="mailto:Swift-devel@ci.uchicago.edu" target="_blank">Swift-devel@ci.uchicago.edu</a><br>
> > > <a href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel" target="_blank">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a><br>
> ><br>
> ><br>
<br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>