[Swift-user] Re: 3rd party transfers

Allan Espinosa aespinosa at cs.uchicago.edu
Fri Dec 10 10:13:05 CST 2010 

Hi Mike.

Yes.  I had the workflow stagein 1, 10, 40 , 80, 400, 800, 2000, 8000,
30000 files.  The throttles are the same for each run.  Problems
started to occur at around 800 files .

For staging in local files, problems started to occur at 30000 files
where vdl:dostagein hits gpfs too much.

-Allan


2010/12/10 Michael Wilde <wilde at mcs.anl.gov>:
> Allan, did you verify that each remote site you are talking to in this test is functional at low transaction rates using your current sites configuration?
>
> I.e., are you certain that the error below is due to load and not a site-related error?
>
> - Mike
>
>
> ----- Original Message -----
>> I tried to have the tests more synthesized using Mike's catsall
>> workflow staging in ~3 MB data files to 5 OSG sites. Swift seem to
>> handle the transfer well when the originating files are local. But
>> when it starts to use remote file objects, I get all these 3rd party
>> transfer exceptions. my throttle for file transfers is 8 and for file
>> operations is 10.
>>
>> 2010-12-09 18:58:16,700-0600 DEBUG DelegatedFileTransferHandler File
>> transfer with resource remote->tmp
>> 2010-12-09 18:58:16,734-0600 DEBUG DelegatedFileTransferHandler
>> Exception in transfer
>> org.globus.cog.abstraction.impl.file.IrrecoverableResourceException:
>> Exception in getFile
>> at
>> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
>> FileResource.java:62)
>> at
>> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
>> :401)
>> at
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doSource(DelegatedFil
>> eTransferHandler.java:269)
>> at
>> org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doSource(Cachi
>> ngDelegatedFileTransferHandler.java:59)
>> at
>> org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTran
>> sferHandler.java:486)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.globus.cog.abstraction.impl.file.FileResourceException:
>> Failed to retrieve file information
>> about
>> /projsmall/osg/data/engage/scec/swift_scratch/catsall-20101209-1839-pnazhid6/info/p/cat-p2em5s2k-in
>> fo
>> at
>> org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(AbstractFTP
>> FileResource.java:51)
>> at
>> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
>> java:550)
>> at
>> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java
>> :384)
>> ... 4 more
>> Caused by: org.globus.ftp.exception.ServerException: Server refused
>> performing the request. Custom message
>> : Server refused MLST command (error code 1) [Nested exception
>> message: Custom message: Unexpected reply:
>> 500-Command failed :
>> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
>> 500-System error in stat: No such file or directory
>> 500-A system call failed: No such file or directory
>> 500 End.] [Nested exception is
>> org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
>> message: Une
>> xpected reply: 500-Command failed :
>> globus_gridftp_server_file.c:globus_l_gfs_file_stat:389:
>> 500-System error in stat: No such file or directory
>> 500-A system call failed: No such file or directory
>> 500 End.]
>> at
>> org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java
>> :101)
>> at org.globus.ftp.FTPClient.mlst(FTPClient.java:643)
>> at
>> org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getGridFile(FileResourceImpl.
>> java:546)
>> ... 5 more
>>
>>
>> I may have been stressing the source gridftp server too much (pads)
>> that it cannot handle a throttle of 8 . But at this configuration, I
>> get low transfer performance. When doing direct transfers, I was able
>> to get better transfer rates until i start coking out gpfs at 10k
>> stageins. My throttle for this configurations was 40 for both file
>> transfers and file operations.
>>
>>
>> 2010/12/2 Allan Espinosa <aespinosa at cs.uchicago.edu>:
>> > I have a bunch of 3rd party gridftp transfers. Swift reports around
>> > 10k jobs being in the vdl:stagein at a time. After a while i get a
>> > couple of these errors. Does it look like i'm stressing the gridftp
>> > servers? my throttle.transfers=8
>> >
>> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler
>> > Starting service on gsiftp://gpn-hus
>> > 2010-12-02 02:22:06,008-0600 DEBUG DelegatedFileTransferHandler File
>> > transfer with resource local->r
>> > 2010-12-02 02:22:06,247-0600 DEBUG DelegatedFileTransferHandler
>> > Exception in transfer
>> > org.globus.cog.abstraction.impl.file.FileResourceException
>> >        at
>> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
>> > esource.java:51)
>> >        at
>> >        org.globus.cog.abstraction.impl.file.ftp.AbstractFTPFileResource.translateException(Abstr
>> > esource.java:34)
>> >        at
>> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
>> > eTransferHandler.java:352)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
>> > ngDelegatedFileTransferHandler.java:46)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
>> > andler.java:489)
>> >        at java.lang.Thread.run(Thread.java:619)
>> > Caused by: org.globus.ftp.exception.ServerException: Server refused
>> > performing the request. Custom m
>> > rror code 1) [Nested exception message: Custom message: Unexpected
>> > reply: 451 ocurred during retrie
>> > org.globus.ftp.exception.DataChannelException: setPassive() must
>> > match
>> > store() and setActive() - ret
>> > rror code 2)
>> > org.globus.ftp.exception.DataChannelException: setPassive() must
>> > match
>> > store() and setActive() - ret
>> > rror code 2)
>> >        at
>> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
>> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
>> >        at
>> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
>> > eTransferHandler.java:352)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
>> > ngDelegatedFileTransferHandler.java:46)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
>> > andler.java:489)
>> >        at java.lang.Thread.run(Thread.java:619)
>> > ] [Nested exception is
>> > org.globus.ftp.exception.UnexpectedReplyCodeException: Custom
>> > message: Unexp
>> > : 451 ocurred during retrieve()
>> > org.globus.ftp.exception.DataChannelException: setPassive() must
>> > match
>> > store() and setActive() - ret
>> > rror code 2)
>> > org.globus.ftp.exception.DataChannelException: setPassive() must
>> > match
>> > store() and setActive() - ret
>> > rror code 2)
>> >        at
>> >        org.globus.ftp.extended.GridFTPServerFacade.retrieve(GridFTPServerFacade.java:469)
>> >        at org.globus.ftp.FTPClient.put(FTPClient.java:1294)
>> >        at
>> >        org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.putFile(FileResourceImp
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doDestination(D
>> > eTransferHandler.java:352)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.CachingDelegatedFileTransferHandler.doDestin
>> > ngDelegatedFileTransferHandler.java:46)
>> >        at
>> >        org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFi
>> > andler.java:489)
>> >        at java.lang.Thread.run(Thread.java:619)
>> > ]
>> >        at
>> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
>> >        at
>> >        org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerExceptio
>> >        at
>> >        org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195)
>> >        ... 1 more
>> >

-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>

More information about the Swift-user mailing list