[Swift-user] Re: 3rd party transfers

Michael Wilde wilde at mcs.anl.gov
Fri Dec 10 09:09:15 CST 2010 

Allan, did you verify that each remote site you are talking to in this test is functional at low transaction rates using your current sites configuration?

I.e., are you certain that the error below is due to load and not a site-related error?

- Mike


----- Original Message -----
> I tried to have the tests more synthesized using Mike's catsall
> workflow staging in ~3 MB data files to 5 OSG sites. Swift seem to
> handle the transfer well when the originating files are local. But
> when it starts to use remote file objects, I get all these 3rd party
> transfer exceptions. my throttle for file transfers is 8 and for file
> operations is 10.
> 
> 2010-12-09 18:58:16,700-0600 DEBUG DelegatedFileTransferHandler File
> transfer with resource remote->tmp
> 2010-12-09 18:58:16,734-0600 DEBUG DelegatedFileTransferHandler
> Exception in transfer
> org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> Exception in getFile
> at
> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
> FileResource.java:62)
> at
> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
> :401)
> at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doSource(DelegatedFil
> eTransferHandler.java:269)
> at
> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doSource(Cachi
> ngDelegatedFileTransferHandler.java:59)
> at
> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTran
> sferHandler.java:486)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.globus.cog.abstraction.impl.file.FileResourceException:
> Failed to retrieve file information
> about
> /projsmall/osg/data/engage/scec/swift_scratch/catsall-20101209-1839-pnazhid6/info/p/cat-p2em5s2k-in
> fo
> at
> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
> FileResource.java:51)
> at
> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
> java:550)
> at
> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
> :384)
> ... 4 more
> Caused by: org.globus.ftp.exception.ServerException: Server refused
> performing the request. Custom message
> : Server refused MLST command (error code 1) [Nested exception
> message: Custom message: Unexpected reply:
> 500-Command failed :
> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
> 500-System error in stat: No such file or directory
> 500-A system call failed: No such file or directory
> 500 End.] [Nested exception is
> org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
> message: Une
> xpected reply: 500-Command failed :
> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
> 500-System error in stat: No such file or directory
> 500-A system call failed: No such file or directory
> 500 End.]
> at
> org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java
> :101)
> at org.globus.ftp.FTPClient.mlst(FTPClient.java:643)
> at
> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
> java:546)
> ... 5 more
> 
> 
> I may have been stressing the source gridftp server too much (pads)
> that it cannot handle a throttle of 8 . But at this configuration, I
> get low transfer performance. When doing direct transfers, I was able
> to get better transfer rates until i start coking out gpfs at 10k
> stageins. My throttle for this configurations was 40 for both file
> transfers and file operations.
> 
> 
> 2010/12/2 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> > I have a bunch of 3rd party gridftp transfers. Swift reports around
> > 10k jobs being in the vdl:stagein at a time. After a while i get a
> > couple of these errors. Does it look like i'm stressing the gridftp
> > servers? my throttle.transfers=8
> >
> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler
> > Starting service on gsiftp://gpn-hus
> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler File
> > transfer with resource local->r
> > 2010-12-02 02:22:06,247-0600 DEBUG DelegatedFileTransferHandler
> > Exception in transfer
> > org.globus.cog.abstraction.impl.file.FileResourceException
> >        at
> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> > esource.java:51)
> >        at
> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> > esource.java:34)
> >        at
> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> > eTransferHandler.java:352)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> > ngDelegatedFileTransferHandler.java:46)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> > andler.java:489)
> >        at java.lang.Thread.run(Thread.java:619)
> > Caused by: org.globus.ftp.exception.ServerException: Server refused
> > performing the request. Custom m
> > rror code 1) [Nested exception message: Custom message: Unexpected
> > reply: 451 ocurred during retrie
> > org.globus.ftp.exception.DataChannelException: setPassive() must
> > match
> > store() and setActive() - ret
> > rror code 2)
> > org.globus.ftp.exception.DataChannelException: setPassive() must
> > match
> > store() and setActive() - ret
> > rror code 2)
> >        at
> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
> >        at
> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> > eTransferHandler.java:352)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> > ngDelegatedFileTransferHandler.java:46)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> > andler.java:489)
> >        at java.lang.Thread.run(Thread.java:619)
> > ] [Nested exception is
> > org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
> > message: Unexp
> > : 451 ocurred during retrieve()
> > org.globus.ftp.exception.DataChannelException: setPassive() must
> > match
> > store() and setActive() - ret
> > rror code 2)
> > org.globus.ftp.exception.DataChannelException: setPassive() must
> > match
> > store() and setActive() - ret
> > rror code 2)
> >        at
> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
> >        at
> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> > eTransferHandler.java:352)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> > ngDelegatedFileTransferHandler.java:46)
> >        at
> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> > andler.java:489)
> >        at java.lang.Thread.run(Thread.java:619)
> > ]
> >        at
> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
> >        at
> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
> >        at
> >        org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
> >        ... 1 more
> >
> 
> 
> 
> --
> Allan M. Espinosa <http://amespinosa.wordpress.com>
> PhD student, Computer Science
> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-user

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

More information about the Swift-user mailing list