[Swift-user] Re: 3rd party transfers

Michael Wilde wilde at mcs.anl.gov
Fri Dec 10 10:17:58 CST 2010 

Did you try provider staging, which might be easier to throttle given that the staging endpoints are more under Swift's control?

- MIke

----- Original Message -----
> Hi Mike.
> 
> Yes. I had the workflow stagein 1, 10, 40 , 80, 400, 800, 2000, 8000,
> 30000 files. The throttles are the same for each run. Problems
> started to occur at around 800 files .
> 
> For staging in local files, problems started to occur at 30000 files
> where vdl:dostagein hits gpfs too much.
> 
> -Allan
> 
> 
> 2010/12/10 Michael Wilde <wilde at mcs.anl.gov>:
> > Allan, did you verify that each remote site you are talking to in
> > this test is functional at low transaction rates using your current
> > sites configuration?
> >
> > I.e., are you certain that the error below is due to load and not a
> > site-related error?
> >
> > - Mike
> >
> >
> > ----- Original Message -----
> >> I tried to have the tests more synthesized using Mike's catsall
> >> workflow staging in ~3 MB data files to 5 OSG sites. Swift seem to
> >> handle the transfer well when the originating files are local. But
> >> when it starts to use remote file objects, I get all these 3rd
> >> party
> >> transfer exceptions. my throttle for file transfers is 8 and for
> >> file
> >> operations is 10.
> >>
> >> 2010-12-09 18:58:16,700-0600 DEBUG DelegatedFileTransferHandler
> >> File
> >> transfer with resource remote->tmp
> >> 2010-12-09 18:58:16,734-0600 DEBUG DelegatedFileTransferHandler
> >> Exception in transfer
> >> org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
> >> Exception in getFile
> >> at
> >> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
> >> FileResource.java:62)
> >> at
> >> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
> >> :401)
> >> at
> >> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doSource(DelegatedFil
> >> eTransferHandler.java:269)
> >> at
> >> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doSource(Cachi
> >> ngDelegatedFileTransferHandler.java:59)
> >> at
> >> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTran
> >> sferHandler.java:486)
> >> at java.lang.Thread.run(Thread.java:619)
> >> Caused by:
> >> org.globus.cog.abstraction.impl.file.FileResourceException:
> >> Failed to retrieve file information
> >> about
> >> /projsmall/osg/data/engage/scec/swift_scratch/catsall-20101209-1839-pnazhid6/info/p/cat-p2em5s2k-in
> >> fo
> >> at
> >> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
> >> FileResource.java:51)
> >> at
> >> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
> >> java:550)
> >> at
> >> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
> >> :384)
> >> ... 4 more
> >> Caused by: org.globus.ftp.exception.ServerException: Server refused
> >> performing the request. Custom message
> >> : Server refused MLST command (error code 1) [Nested exception
> >> message: Custom message: Unexpected reply:
> >> 500-Command failed :
> >> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
> >> 500-System error in stat: No such file or directory
> >> 500-A system call failed: No such file or directory
> >> 500 End.] [Nested exception is
> >> org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
> >> message: Une
> >> xpected reply: 500-Command failed :
> >> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
> >> 500-System error in stat: No such file or directory
> >> 500-A system call failed: No such file or directory
> >> 500 End.]
> >> at
> >> org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java
> >> :101)
> >> at org.globus.ftp.FTPClient.mlst(FTPClient.java:643)
> >> at
> >> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
> >> java:546)
> >> ... 5 more
> >>
> >>
> >> I may have been stressing the source gridftp server too much (pads)
> >> that it cannot handle a throttle of 8 . But at this configuration,
> >> I
> >> get low transfer performance. When doing direct transfers, I was
> >> able
> >> to get better transfer rates until i start coking out gpfs at 10k
> >> stageins. My throttle for this configurations was 40 for both file
> >> transfers and file operations.
> >>
> >>
> >> 2010/12/2 Allan Espinosa <aespinosa at cs.uchicago.edu>:
> >> > I have a bunch of 3rd party gridftp transfers. Swift reports
> >> > around
> >> > 10k jobs being in the vdl:stagein at a time. After a while i get
> >> > a
> >> > couple of these errors. Does it look like i'm stressing the
> >> > gridftp
> >> > servers? my throttle.transfers=8
> >> >
> >> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler
> >> > Starting service on gsiftp://gpn-hus
> >> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler
> >> > File
> >> > transfer with resource local->r
> >> > 2010-12-02 02:22:06,247-0600 DEBUG DelegatedFileTransferHandler
> >> > Exception in transfer
> >> > org.globus.cog.abstraction.impl.file.FileResourceException
> >> >        at
> >> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> >> > esource.java:51)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
> >> > esource.java:34)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> >> > eTransferHandler.java:352)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> >> > ngDelegatedFileTransferHandler.java:46)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> >> > andler.java:489)
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> > Caused by: org.globus.ftp.exception.ServerException: Server
> >> > refused
> >> > performing the request. Custom m
> >> > rror code 1) [Nested exception message: Custom message:
> >> > Unexpected
> >> > reply: 451 ocurred during retrie
> >> > org.globus.ftp.exception.DataChannelException: setPassive() must
> >> > match
> >> > store() and setActive() - ret
> >> > rror code 2)
> >> > org.globus.ftp.exception.DataChannelException: setPassive() must
> >> > match
> >> > store() and setActive() - ret
> >> > rror code 2)
> >> >        at
> >> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
> >> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> >> > eTransferHandler.java:352)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> >> > ngDelegatedFileTransferHandler.java:46)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> >> > andler.java:489)
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> > ] [Nested exception is
> >> > org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
> >> > message: Unexp
> >> > : 451 ocurred during retrieve()
> >> > org.globus.ftp.exception.DataChannelException: setPassive() must
> >> > match
> >> > store() and setActive() - ret
> >> > rror code 2)
> >> > org.globus.ftp.exception.DataChannelException: setPassive() must
> >> > match
> >> > store() and setActive() - ret
> >> > rror code 2)
> >> >        at
> >> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
> >> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
> >> > eTransferHandler.java:352)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
> >> > ngDelegatedFileTransferHandler.java:46)
> >> >        at
> >> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
> >> > andler.java:489)
> >> >        at java.lang.Thread.run(Thread.java:619)
> >> > ]
> >> >        at
> >> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
> >> >        at
> >> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
> >> >        at
> >> >        org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
> >> >        ... 1 more
> >> >
> 
> --
> Allan M. Espinosa <http://amespinosa.wordpress.com>
> PhD student, Computer Science
> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

More information about the Swift-user mailing list