[Swift-user] Re: 3rd party transfers

Allan Espinosa aespinosa at cs.uchicago.edu
Thu Dec 9 21:01:08 CST 2010 

I tried to have the tests more synthesized using Mike's catsall
workflow staging in ~3 MB data files to 5 OSG sites.  Swift seem to
handle the transfer well when the originating files are local.  But
when it starts to use remote file objects, I get all these 3rd party
transfer exceptions.  my throttle for file transfers is 8 and for file
operations is 10.

2010-12-09 18:58:16,700-0600 DEBUG DelegatedFileTransferHandler File
transfer with resource remote->tmp
2010-12-09 18:58:16,734-0600 DEBUG DelegatedFileTransferHandler
Exception in transfer
org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
Exception in getFile
        at org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
FileResource.java:62)
        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
:401)
        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doSource(DelegatedFil
eTransferHandler.java:269)
        at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doSource(Cachi
ngDelegatedFileTransferHandler.java:59)
        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTran
sferHandler.java:486)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.globus.cog.abstraction.impl.file.FileResourceException:
Failed to retrieve file information
 about /projsmall/osg/data/engage/scec/swift_scratch/catsall-20101209-1839-pnazhid6/info/p/cat-p2em5s2k-in
fo
        at org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
FileResource.java:51)
        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
java:550)
        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
:384)
        ... 4 more
Caused by: org.globus.ftp.exception.ServerException: Server refused
performing the request. Custom message
: Server refused MLST command (error code 1) [Nested exception
message:  Custom message: Unexpected reply:
 500-Command failed : globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
500-System error in stat: No such file or directory
500-A system call failed: No such file or directory
500 End.] [Nested exception is
org.globus.ftp.exception.UnexpectedReplyCodeException:  Custom
message: Une
xpected reply: 500-Command failed :
globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
500-System error in stat: No such file or directory
500-A system call failed: No such file or directory
500 End.]
        at org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java
:101)
        at org.globus.ftp.FTPClient.mlst(FTPClient.java:643)
        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
java:546)
        ... 5 more


I may have been stressing the source gridftp server too much (pads)
that it cannot handle a throttle of 8 .  But at this configuration, I
get low transfer performance.  When doing direct transfers, I was able
to get better transfer rates until i start coking out gpfs at 10k
stageins.  My throttle for this configurations was 40 for both file
transfers and file operations.


2010/12/2 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> I have  a bunch of 3rd party gridftp transfers.   Swift reports around
> 10k jobs being in the vdl:stagein at a time.  After a while i get a
> couple of these errors.  Does it look like i'm stressing the gridftp
> servers? my throttle.transfers=8
>
> 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler
> Starting service on gsiftp://gpn-hus
> 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler File
> transfer with resource local->r
> 2010-12-02 02:22:06,247-0600 DEBUG DelegatedFileTransferHandler
> Exception in transfer
> org.globus.cog.abstraction.impl.file.FileResourceException
>        at org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> esource.java:51)
>        at org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> esource.java:34)
>        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> eTransferHandler.java:352)
>        at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> ngDelegatedFileTransferHandler.java:46)
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> andler.java:489)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: org.globus.ftp.exception.ServerException: Server refused
> performing the request. Custom m
> rror code 1) [Nested exception message:  Custom message: Unexpected
> reply: 451 ocurred during retrie
> org.globus.ftp.exception.DataChannelException: setPassive() must match
> store() and setActive() - ret
> rror code 2)
> org.globus.ftp.exception.DataChannelException: setPassive() must match
> store() and setActive() - ret
> rror code 2)
>        at org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
>        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
>        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> eTransferHandler.java:352)
>        at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> ngDelegatedFileTransferHandler.java:46)
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> andler.java:489)
>        at java.lang.Thread.run(Thread.java:619)
> ] [Nested exception is
> org.globus.ftp.exception.UnexpectedReplyCodeException:  Custom
> message: Unexp
> : 451 ocurred during retrieve()
> org.globus.ftp.exception.DataChannelException: setPassive() must match
> store() and setActive() - ret
> rror code 2)
> org.globus.ftp.exception.DataChannelException: setPassive() must match
> store() and setActive() - ret
> rror code 2)
>        at org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
>        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
>        at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> eTransferHandler.java:352)
>        at org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> ngDelegatedFileTransferHandler.java:46)
>        at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> andler.java:489)
>        at java.lang.Thread.run(Thread.java:619)
> ]
>        at org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
>        at org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
>        at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
>        ... 1 more
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>

More information about the Swift-user mailing list